Efficient image capturing based on eyelid position

ABSTRACT

Systems, methods, and computer programs for eye tracking involving a camera that is operated according to eyelid position. In one embodiment, an inward-facing head-mounted camera captures images of an eye. A head-mounted device emits electromagnetic waves towards the eye and measures reflections of these waves from the eye. The average rate of reflection measurements is at least ten times the average rate of image capture. A computer detects the position of at least one of the eyelids covering the eye based on the reflection measurements. And the camera operation is adjusted according to the eyelid position in a manner that decreases, in the captured images, areas depicting the eyelids, and optionally increases areas depicting the iris.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application is a Continuation of U.S. application Ser. No.17/490,432, filed Sep. 30, 2021. U.S. application Ser. No. 17/490,432claims priority to U.S. Provisional Patent Application No. 63/113,846,filed Nov. 14, 2020, U.S. Provisional Patent Application No. 63/122,961,filed Dec. 9, 2020, and U.S. Provisional Patent Application No.63/140,453 filed Jan. 22, 2021

BACKGROUND

Eye tracking using video-oculography (VOG) is important for theperformance of many head-mounted systems, such as mixed realitydisplays. For example, VOG can enable more efficient image rendering(through foveated rendering), as well as determining various pupilfeatures that may be indicative of the user's physiological andemotional states. The performance of VOG, and the utility frominformation obtained using VOG, can be highly dependent on the qualityof images of the eye that are captured, and in particular, on whetherthe user blinking at the time (which can lead to occlusion of the pupilin the captured images). The rate of typical eye blinking variesaccording to the person and the task performed, and is on averagebetween of 4 and 17 blinks/minute, with an average duration of a blinkbeing between 100 ms and 400 ms. Thus, during a significant amount oftime, eye blinking can render images of the eye to be less useful foreye tracking.

Eye tracking using VOG can require expenditure of significant power bybattery-operated head-mounted systems. But typical VOG systems do nottake into account blinking when capturing images and/or processingcaptured images. Thus, there is a need to be able to account forblinking when performing eye tracking in order to save power withbattery-operated systems.

SUMMARY

Eye blinking affects the performance of video-based eye trackers thatcannot track the limbus and/or the pupil when occluded by the eyelid. Inaddition, when the eye is partially occluded by the eyelid, at thebeginning and end of the blink, gaze point estimation accuracy maydecrease.

Video cameras are not the only way to obtain data useful for eyetracking. Various types of head-mounted devices can provide measurementsthat are indicative of the eye's position and/or the eyelid's position.For example, optical flow sensors, range sensors, andphotosensor-oculography devices (PSOG) can also provide informationabout the eye and/or eyelid's position. These types of devices aretypically more power-efficient than VOG, but also typically provide lessdetailed and/or less accurate information about the eye and/or eyelid.

Some aspects of this disclosure involve eye tracking that utilizesinformation about the eyelid position in order to operate in a moreefficient manner a head-mounted camera that is used for VOG. In someembodiments, a head-mounted device is utilized to detect eyelid positionand then this information is used for timing the image capturing, and/oradjusting the size and/or region of the eye captured in the images, toinclude the visible portion of the eye. Because battery operatedhead-mounted systems usually strive to save power, and the head-mounteddevices disclosed in embodiments described herein (e.g., PSOG, rangesensors, etc.) may be more power efficient than VOG, using suchhead-mounted devices along with VOG, as described herein, can save powerwhen performing eye tracking.

One aspect of this disclosure involves a system that operates a cameraaccording to eyelid position. In one embodiment, the system includes aninward-facing head-mounted camera that captures images of an eye, ahead-mounted device, and a computer. The head-mounted device emitselectromagnetic waves towards the eye, and take measurements ofreflections of the electromagnetic waves from the eye. Optionally, anaverage rate at which the measurements of the reflections are taken isat least ten times an average rate at which the images are captured. Thecomputer detects, based on the measurements of the reflections, aposition of at least one of the eyelids covering the eye (eyelidposition). The computer operates the inward-facing head-mounted cameraaccording to the eyelid position in a manner that decreases, in thecaptured images, areas depicting the at least one of the eyelids.

In one example, the head-mounted device comprises aphotosensor-oculography device (PSOG) that measures reflections of lightemitted by the PSOG towards the eye. In another example, thehead-mounted device comprises an optical flow sensor that measuresoptical flow and/or visual motion, and the eyelid position is calculatedbased on an optical flow algorithm. In yet another example, thehead-mounted device comprises range sensors that measure ranges betweenthe range sensors and the eye (measurements of the range sensors areaffected by the eyelid position).

In one embodiment, the inward-facing head-mounted camera utilizes asensor that supports changing of its region of interest (ROI), and thecomputer utilizes the eyelid position to set the ROI around a portion ofthe eye that is not covered by the at least one of the eyelids.Optionally, the ROI covers less than half the sensor's field of view.Optionally, the computer sets the ROI such that the size of the ROI isproportional to area of the eyeball uncovered by the eyelid. Optionally,the computer detects a raising of the upper eyelid based on themeasurements of the reflections, and increases a frame rate of theinward-facing head-mounted camera during the time the upper eyelid israised above a threshold. Optionally, the computer detects a raising ofthe lower eyelid based on the measurements of the reflections, andincreases a frame rate of the inward-facing head-mounted camera duringthe time the lower eyelid is raised above a threshold.

In one embodiment, in order to operate the inward-facing head-mountedcamera according to the eyelid position the computer refrains fromcommanding the camera to capture the images during blinks and/or whilethe eye is closed.

In one embodiment, the computer extracts eyelid positions by analyzingthe images (image-based positions), and utilizes the image-basedpositions and the measurements of the reflections to generate labeledtraining samples for training a machine learning-based model fordetecting the eyelid position based on additional measurements ofreflections taken with the head-mounted device.

In another embodiment, the computer receives photos from a camerabelonging to a non-head-mounted device, extracts eyelid positions byanalyzing the photos (photo-based positions), and utilizes thephoto-based positions and the measurements of the reflections togenerate labeled training samples for training a machine learning-basedmodel for detecting the eyelid position based on additional measurementsof reflections taken with the head-mounted device. Optionally, thecomputer comprises a head-mounted computer and a non-head-mountedcomputer that communicate over a wireless communication channel, thenon-head-mounted computer trains the model, and the head-mountedcomputer uses the trained model for detecting the eyelid position basedon reflections.

Another aspect of this disclosure involves a method that includes atleast the following steps: capturing images of an eye by aninward-facing head-mounted camera; emitting, by a head-mounted device,electromagnetic waves towards the eye, and taking measurements ofreflections of the electromagnetic waves from the eye, where an averagerate of taking measurements of the reflections is at least ten times anaverage rate of capturing the images; detecting, based on themeasurements of the reflections, a position of at least one of theeyelids covering the eye (eyelid position); and operating theinward-facing head-mounted camera according to the eyelid position in amanner that decreases, in the captured images, areas depicting the atleast one of the eyelids.

In one embodiment, the inward-facing head-mounted camera utilizes asensor that supports changing of its region of interest (ROI), and themethod includes a step of utilizing the eyelid position to set the ROIaround a portion of the eye that is not covered by the at least one ofthe eyelids.

In one embodiment, the method optionally involves performing thefollowing steps to train a machine learning-based model that is used fordetecting the eyelid position based on additional measurements ofreflections: extracting eyelid positions by analyzing the images(image-based positions); generating labels based on the image-basedpositions; generating feature values based on the measurements of thereflections; and providing the labels and feature values to train themachine learning-based model for detecting the eyelid position based onthe additional measurements of the reflections.

In one embodiment, the method optionally involves performing thefollowing steps to train a machine learning-based model that is used fordetecting the eyelid position based on additional measurements ofreflections: receiving photos from a camera belonging to anon-head-mounted device, extracting eyelid positions by analyzing thephotos (photo-based positions), generating labels based on thephoto-based positions, generating feature values based on themeasurements of the reflections, and providing the labels and featurevalues to train the machine learning-based model for detecting theeyelid position based on the additional measurements of the reflections.

Yet another aspect of this disclosure involves a non-transitory computerreadable medium storing one or more computer programs configured tocause a processor-based system to execute steps of one or moreembodiments of the aforementioned method.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are herein described by way of example only, withreference to the following drawings:

FIG. 1 illustrates an embodiment of an eye tracking system that includesa photosensor-oculography device (PSOG) and a camera;

FIG. 2A illustrates smartglasses with PSOG that utilizes multiple lightsources and multiple detectors;

FIG. 2B illustrates an embodiment of an eye tracking system onsmartglasses that tracks both eyes;

FIG. 3 illustrates eye tracking utilizing head-mounted PSOG and anon-head-mounted camera;

FIG. 4 illustrates an embodiment of an eye tracking system that utilizesan event camera;

FIG. 5 illustrates an embodiment of an eye tracking system that includesPSOG and a camera that utilize a sensor that supports changing of itsregion of interest (ROI);

FIG. 6 illustrates a scenario in which different ROIs are read when theeye is at different eye positions;

FIG. 7 illustrates an embodiment of an eye tracking system that utilizesmeasurements indicative of the eye position to set an ROI of a camerathat captures images of the eye;

FIG. 8 illustrates an embodiment of an eye tracking system that utilizesmeasurements indicative of the eye position to crop an image of the eye;

FIG. 9A illustrates an embodiment of an eye tracking system thatoperates a camera that captures images of the eye based on eyepositions;

FIG. 9B illustrates an embodiment in which multiple head-mounted camerasare used for eye tracking;

FIG. 10 illustrates an embodiment of an eye tracking system thatinvolves operation of a camera according to eyelid position;

FIG. 11A illustrates different ROIs that are set according to differenteyelid positions;

FIG. 11B illustrates selective issuing of commands to capture imagesbased on the extent to which the eye is open; and

FIG. 12A and FIG. 12B are schematic illustrations of possibleembodiments for computers.

DETAILED DESCRIPTION

The term “oculography” as used herein refers to measuring eye positionand/or eye movements of either one eye or both eyes (i.e., monocular orbinocular measurements). When supported by the system, oculography mayalso measure other eye parameters, such as pupil diameter and/or extentof blinking.

The term “eye tracking” as used herein refers to measuring eye position,eye movement, and/or gaze direction. An “eye tracker” is a device thatmeasures and/or calculates values indicative of eye position, eyemovement, and/or gaze direction. Some embodiments of an “eye trackingsystem” described herein may be considered an eye tracker, a componentof an eye tracker, or a component that provides data utilized by an eyetracker. When supported, an eye tracking system may measure other eyeparameters, such as pupil diameter and/or extent of blinking.

As used herein, the terms “photosensor-oculography” and“photosensor-oculography device” (referred to herein as “PSOG”), referto measuring eye position and/or eye movements (or equivalents thereof)and/or of a device used for said measuring, based on the context.Measurements of the eye position and/or the eye movements, may bemeasurements of either of one eye or both eyes (depending on thecontext) Taking such measurements is done based on the principle ofemitting light and capturing the reflected light from the user's eyesurface and/or eyelid with discrete photosensors. Alternative termsknown in the art for PSOG include PS-OG, photoelectric-technique,infrared-oculography, infrared-reflectance-oculography,limbus-reflection-technique, and eye tracking based on steering a beamthrough MEMS. The following reference, which is referred to herein asRigas 2018, and which is incorporated herein by reference, provides areview of PSOG techniques: Rigas, Ioannis, Hayes Raffle, and Oleg V.Komogortsev “Photosensor oculography: Survey and parametric analysis ofdesigns using model-based simulation” IEEE Transactions on Human-MachineSystems 99 (2018): 1-12.

The term “discrete photosensors” refers to very-low resolution lightdetectors that are relatively low cost and low power, such asphotosensitive sensors, photodetectors, photodiodes, Light EmittingDiodes (LEDs) having a bi-directional characteristic with the ability toemit the light and to measure reflections, single detectors, splitdetectors, four-quadrant detectors, position-sensitive detectors, photoreflective sensors (for modules combining both the emitter andreceiver), sensors with less than a thousand sensing pixels on the samesubstrate (i.e., the term discrete photosensor is not limited to asingle-pixel photosensor), and arrays with direct wire connections toeach pixel supporting parallel readout. The definition of discretephotosensors explicitly excludes camera sensors havingthousands/millions of pixels that are equipped with suitable optics forso many pixels, such as CCD and CMOS video camera sensors havingthousands/millions of pixels.

PSOG, as defined herein, may also be implemented using a scanning lightbeam. For example, U.S. Pat. No. 10,317,672 and U.S. patent applications2018/0210547, 2019/0204912, and 2019/0204913 to AdHawk Microsystemsdescribe a version of PSOG that is based on steering a light beam usinga microelectromechanical system (MEMS) onto a surface of the eye, anddetecting light reflected from features of the eye with one or morephotosensors. In another example, US patent application 2020/0285307 toMicrosoft Technology Licensing LLC describes another version of PSOGthat is based on steering a light beam using MEMS and through a displaymodule assembly onto the eye, and detecting the reflections with one ormore photodetectors. In still another example, the thesis of Zhao,Hongfan “Micro-Scanning Mirror based Eye-tracking Technology” (2020),describes still another version of PSOG that is based on ascanning-micro-minor to scan a laser beam on the eyeball, and a lineararray photodetector to detect the light reflected from the eyeball.

Sentences in the form of “to calculate Y based on the reflectionsmeasured by the PSOG” or “to calculate Y based on measurements ofreflections (taken by the PSOG)” are to be interpreted in the context ofthe structure and operation of the PSOG. For example, when the PSOGincludes multiple light sources and multiple discrete photosensors, thereflections are indicative of which light source is emitting at a giventime and intensities of the corresponding measured reflections due tothose emissions. In a second example, the PSOG includes a scanning lightbeam and multiple discrete photosensors, and the reflections areindicative of the directions of the scanning light beam and thecorresponding reflections measured by the photosensors. In a thirdexample, the reflections are indicative of one or more of the followingeye parameters calculated by the PSOG: eye position, eye movement, eyemovement velocity, upper and/or lower eyelid position, and blinking.

Measurements of the reflections may be expressed using various units, indifferent embodiments. In some embodiments, the measurements of thereflections may be the raw output of the photosensors expressed asvalues of voltage or illuminance (e.g., expressed as lux). In someembodiments, the measurements of the reflections may undergo variouspreprocessing and/or filtering using techniques known in the art.

The term “video-oculography” (VOG), as used herein, refers to measuringeye position and/or eye movements (or equivalents thereof), of eitherone eye or both eyes, based on processing of images captured by one ormore video cameras. Alternative terms known in the art forvideo-oculography include “video-based eye-tracker”, “video-basedeye-gaze tracking”, “Infrared OcculoGraphy”, “video-nystagmography”, and“infrared video in electronystagmography”. The following references,which are incorporated herein by reference, provide reviews ofvideo-oculography techniques and their usages: (i) Tonsen, Marc, ChrisKay Baumann, and Kai Dierkes “A High-Level Description and PerformanceEvaluation of Pupil Invisible” arXiv preprint arXiv:2009.00508 (2020),(ii) Cristina, Stefania, and Kenneth P. Camilleri “Unobtrusive andpervasive video-based eye-gaze tracking” Image and Vision Computing 74(2018): 21-40, and (iii) Brunyé, Tad T., et al. “A review of eyetracking for understanding and improving diagnostic interpretation”Cognitive research: principles and implications 4.1 (2019): 1-16.

In some embodiments, a camera used for VOG captures intensities, such asthe cameras used by the above VOG references, and the images captured bythe camera represent the intensities measured by the pixels.

In other embodiments, a camera used for VOG may be an event camera (alsoknown as a neuromorphic camera, silicon retina, or dynamic visionsensor) that outputs data that comprises pixel-level brightness changes.In these embodiments, the pixels of the event camera may operateindependently and asynchronously, and report changes in brightness asthey occur.

When a camera used in VOG is an event camera, the word “images” insentences like “a camera configured to capture images (of the eye)”refers to data captured by the camera, which may be images (e.g.,frames) and/or data that may be converted to images and/or equivalentsthereof for the purpose of additional computations (e.g., by serving asinput for machine learning-based methods). In one embodiment, the imagesare reconstructed based on events reported by the event camera, such asthe approach disclosed by U.S. Pat. No. 10,466,779 that is incorporatedherein by reference. In another embodiment, events captured by an eventcamera may be converted into synchronous dense, image-likerepresentations, which can be processed by traditional machine learningmethods developed for standard cameras. Such an approach is disclosed byMessikommer, Nico, et al. “Event-based Asynchronous Sparse ConvolutionalNetworks” arXiv preprint arXiv:2003.09148 (2020). In yet anotherembodiment, an event camera may capture both frames and events. Thus, areference to “images” in such cases may refer both frames of intensities(e.g., acquired through synchronous reading of photodiodes) andasynchronous events. This type of images is disclosed by U.S. Pat. No.10,466,779 and by the reference Angelopoulos, Anastasios N., et al.“Event Based, Near Eye Gaze Tracking Beyond 10,000 Hz” arXiv preprintarXiv:2004.03577 (2020), which are incorporated herein by reference.

Various embodiments described herein involve a head-mounted system (HMS)that includes VOG and/or PSOG. In these embodiments, the VOG and/or PSOGinclude one or more light emitting components and/or one or more sensingcomponents that may be part of the HMS.

Some examples of the one or more emitting components that may beutilized by systems that include VOG and/or PSOG (such as theaforementioned HMS) include various types of LED or laser emitters.Optionally, these devices are coupled to a head-mounted system atvarious positions suitable to emit light to the wearer's eyes. In oneexample, emitters may be embedded in a head-mounted frame. In anotherexample, emitters may be embedded in the smartglasses' temples. In yetanother example, emitters may be embedded in a display (e.g., a microLEDdisplay with IR emitters located in front of an eye, such as disclosedin U.S. patent application 2020/0335032). In still another example,emitters may be configured to direct their light to the eye through awaveguide (e.g., emitters coupled to an augmented reality display modulewaveguide located in front of the eyes, such as disclosed in U.S. patentapplication 2020/0285307).

Some examples of the one or more sensing components that may be utilizedby systems that include VOG and/or PSOG (such as the aforementioned HMS)include various types of photosensors (e.g., discrete photosensors orimaging sensors of cameras). In one example, photosensors may beembedded in a head-mounted frame. In another example, photosensors maybe embedded in smartglasses' temples. In yet another example,photosensors may be embedded in a display located in front of an eye).In still another example, photosensors may be configured to receive thereflected light from a waveguide (e.g., photosensors coupled to anaugmented reality display module waveguide located in front of theeyes).

In various embodiments disclosed herein VOG may be used in conjunctionwith PSOG. For example, an HMS may include one or more components thatenable VOG and also one or more additional components that enable PSOG.There may be various reasons and motivations for having PSOG along withVOG in some embodiments of eye tracking systems, such as saving power,improving accuracy of eye tracking, and/or extending the range oftrackable eye positions, as discussed below.

Due to the different nature of the signals being analyzed, VOG istypically more accurate than PSOG when it comes to the task of eyetracking (e.g., VOG will often make more accurate determinations of eyepositions than PSOG). However, when tracking the eye positions at thesame rate, PSOG typically consumes significantly less power than VOG,and thus PSOG is considered low-power relative of VOG. The referenceRigas, Ioannis, Hayes Raffle, and Oleg V. Komogortsev “Hybrid ps-vtechnique: A novel sensor fusion approach for fast mobile eye-trackingwith sensor-shift aware correction” IEEE Sensors Journal 17.24 (2017):8356-8366, (referred to herein as Rigas 2017, and is incorporated hereinby reference), discloses a system combining VOG and PSOG, which is ableto calibrate sensor shifts affecting the PSOG models based on theaccurate results of the VOG.

Although the hybrid eye tracker of Rigas 2017 reduced the powerconsumption significantly, the reduction is not sufficient for alightweight HMS, and as a result both the authors of Rigas 2017, and allthe other authors of the publications citing Rigas 2017, directed theirefforts towards developing a standalone PSOG (without VOG) that isrobust to sensor shifts. Three examples of the current researchdirections following Rigas 2017 include (i) Zemblys, Raimondas, and OlegKomogortsev “Making stand-alone PS-OG technology tolerant to theequipment shifts” Proceedings of the 7th Workshop on Pervasive EyeTracking and Mobile Eye-Based Interaction, 2018, (ii) Katrychuk, Dmytro,Henry K. Griffith, and Oleg V. Komogortsev “Power-efficient andshift-robust eye-tracking sensor for portable VR headsets” Proceedingsof the 11th ACM Symposium on Eye Tracking Research & Applications, 2019,and (iii) Li, Richard, et al. “Optical Gaze Tracking withSpatially-Sparse Single-Pixel Detectors” arXiv preprint arXiv:2009.06875(2020). However, the inventors of this invention do not overwhelminglyagree with the current trend of aspiring to a standalone PSOG (which isalso taken by the above cited references discussing PSOG with MEMSscanning beam). The inventors believe it is better to further optimizethe power consumption and/or the hardware architecture and operation ofthe combined VOG and PSOG system, as described below.

Some of the embodiments disclosed herein may combine VOG and PSOG. Thiscombination may be done in various ways. In some embodiments, PSOG maybe operated essentially independently from the VOG. Optionally, in theseembodiments, the high-rate, low-power PSOG measurements may be used tooptimize the performance of the lower-rate, higher-power VOG. In otherembodiments, the more accurate VOG may be used to correct errors in eyeparameters derived from PSOG, while optimizing the operation of the VOGbased on the PSOG measurements, in order to improve the performances ofknown methods (such as described in Rigas 2017) to make them suitablefor a wireless HMS.

In one embodiment, a sensor fusion algorithm is utilized to combine VOGand PSOG measurements. This algorithm includes the following steps:

In step 1, the computer receives images from the head-mounted camera,and determines the eye position based on at least one of the images. Thecomputer may also determine gaze based on images of both eyes. The gazemay be relative to the frame used to mount the cameras, relative to theuser's head, and/or relative to other suitable coordinates known in theart.

In step 2, the computer calculates one or more values related to the eyebased on measurements of reflections taken by the PSOG. In one example,the measurements of the reflections include identification of the lightsource emitting the light and intensities of the corresponding measuredreflections measured by the PSOG. In this step, the computer calculatesone or more values related to the eye, such as eye position, eyemovement, and/or eye movement velocity. Usually, the PSOG measurementshave lower latency and lower accuracy compared to the VOG measurements.

And in step 3, the computer combines the VOG and PSOG measurements forat least one of (i) calibrating the PSOG models based on the VOGresults, and (ii) improving the accuracy of the higher-frequencylower-accuracy PSOG results based on the lower-frequency higher-accuracyVOG results. Examples of known algorithms, useful for sensor fusion thatcombines the VOG and PSOG measurements, include Kalman filter, Bayesiannetwork, and convolutional neural network. The combined measurements mayalso be used to predict future values related to the eye, such as afuture eye position or a future gaze direction. In some examples, thesefuture values are used for foveated rendering, for setting the camera'swindowing parameters, for setting the camera's timings of capturingimages, and/or for setting the camera's smart binning parameters.

Various embodiments described herein involve calculations based onmachine learning approaches. Herein, the terms “machine learningapproach” and/or “machine learning-based approaches” refer to learningfrom examples using one or more approaches. Examples of machine learningapproaches include: decision tree learning, association rule learning,regression models, nearest neighbors classifiers, artificial neuralnetworks, deep learning, inductive logic programming, support vectormachines, clustering, Bayesian networks, reinforcement learning,representation learning, similarity and metric learning, sparsedictionary learning, genetic algorithms, rule-based machine learning,and/or learning classifier systems. Herein, a “machine learning-basedmodel” is a model trained using one or more machine learning approaches.

Herein, “feature values” (also known as feature vector, feature data,numerical features, and inputs) may be considered input to a computerthat utilizes a model to perform the calculation of a value (e.g., anoutput, “target value”, or label) based on the input. It is to be notedthat the terms “feature” and “feature value” may be used interchangeablywhen the context of their use is clear. However, a “feature” typicallyrefers to a certain type of value, and represents a property, while“feature value” is the value of the property with a certain instance(i.e., the value of the feature in a certain sample).

In addition to feature values generated based on measurements taken bysensors mentioned in a specific embodiment, at least some feature valuesutilized by a computer of the specific embodiment may be generated basedon additional sources of data that were not specifically mentioned inthe specific embodiment. Some examples of such additional sources ofdata include: contextual information, information about the user being,measurements of the environment, and values of physiological signals ofthe user obtained by other sensors.

Sentences in the form of “inward-facing head-mounted camera” refer to acamera configured to be worn on a user's head and to remain pointed atthe region it captures (sometimes referred to as ROI), which is on theuser's face, also when the user's head makes angular and lateralmovements. A head-mounted camera (which may be inward-facing and/oroutward-facing) may be physically coupled to a frame worn on the user'shead, may be physically coupled to eyeglasses using a clip-on mechanism(configured to be attached to and detached from the eyeglasses), may bephysically coupled to a hat or a helmet, or may be mounted to the user'shead using any other known device that keeps the camera in a fixedposition relative to the user's head.

The term “smartglasses” refers to any type of a device that resembleseyeglasses, which includes a frame configured to be worn on a user'shead and electronics to operate one or more sensors.

The term “visible-light camera” refers to a non-contact device designedto detect at least some of the visible spectrum, such as a video camerawith optical lenses and CMOS or CCD sensor; visible-light camera may besensitive to near-infrared wavelengths below 1050 nanometer. The term“thermal camera” refers to a non-contact device that measureselectromagnetic radiation having wavelengths longer than 2500 nanometer(nm) and does not touch the region it measures. A thermal camera mayinclude one sensing element (pixel), or multiple sensing elements thatare also referred to herein as “sensing pixels”, “pixels”, and/orfocal-plane array (FPA). A thermal camera may be based on an uncooledthermal sensor, such as a thermopile sensor, a microbolometer sensor(where microbolometer refers to any type of a bolometer sensor and itsequivalents), a pyroelectric sensor, or a ferroelectric sensor.

A reference to a “camera” herein may relate to various types of devices.In one example, a camera may be a visible-light camera. In anotherexample, a camera may capture light in the ultra-violet range. Inanother example, a camera may capture near-infrared radiation (e.g.,wavelengths between 750 and 2000 nm). And in still another example, acamera may be a thermal camera.

The term “temperature sensor” refers to a device that measurestemperature and/or temperature change. The temperature sensor may be acontact thermometer (such as a thermistor, a thermocouple), and/or anon-contact thermal cameras (such as a thermopile sensor, amicrobolometer sensor, or a cooled infrared sensor). Some examples oftemperature sensors useful to measure skin temperature include:thermistors, thermocouples, thermoelectic effect, thermopiles,microbolometers, and pyroelectric sensors. Some examples of temperaturesensors useful to measure environment temperature include: thermistors,resistance temperature detectors, thermocouples; thermopiles, andsemiconductor-based sensors.

The term “movement sensor” refers to a sensor comprising one or more ofthe following components: a 3-axis gyroscope, a 3-axis accelerometer,and a magnetometer. The movement sensor may also include a sensor thatmeasures barometric pressure.

The term “acoustic sensor” refers to a device that converts sound wavesinto an electrical signal. The acoustic sensor may be a microphone, suchas a dynamic microphone, a piezoelectric microphone, a fiber-opticmicrophone, a Micro-Electrical-Mechanical System (MEMS) microphone,and/or other known sensors that measure sound waves.

FIG. 1 illustrates an embodiment of an eye tracking system. Embodimentsof this system include a PSOG 235 that emits light and measuresreflections of the light from an eye of a user, a camera 237 thatcaptures images 239 of the eye of the user, and a computer 240 thatutilizes measurements 236 of the reflections and the images 239 toperform operations involved in tracking the eye of the user. It is notedthat a phrase such as “images of the eye” is to be interpreted as imagesof at least portions of the eye and/or the eyelid.

In some embodiments, the PSOG 235 and the camera 237 are both coupled toa frame of smartglasses 230. In FIG. 1 , as well as illustrations ofother systems in this disclosure that include devices (e.g., PSOG,cameras, etc.) that are coupled to frames of smartglasses, a dotted lineconnecting a component to the frames, such as the lines in FIG. 1between the PSOG 235 and the smartglasses 230 or the line between thecamera 237 the smartglasses 230, are meant to indicate that thecomponents are coupled to frames. This line is for illustration purposesonly, and is meant to illustrate the fact that the components arecoupled to the frames. The line is not meant to indicate a specificlocation on the frames at which the components are to be coupled and/ora nature of the coupling.

It is to be noted that embodiments described herein that include PSOGand/or cameras that are utilized for eye tracking may include multiplePSOG and cameras. For example, the smartglasses 230 may include one ormore PSOGs, in addition to the PSOG 235, which may be utilized to trackthe same eye of the user that is tracked utilizing the PSOG 235, or totrack the user's other eye. Additionally, implementations PSOG used inembodiments described herein (such as the PSOG 235) may involveutilization of various combinations of emitters and photosensordetectors. Some examples of combinations of different numbers/locationsof emitters and/or photosensor detectors are discussed in referencesmentioned herein (e.g., Rigas 2018).

In one embodiment, the computer 240 is configured to calculate valuesindicative of eye movement velocity (EMV) based on the measurements 236of the reflections, and utilize the values to determine settings 238that control how data is read from the camera 237. Optionally, thecomputer 240 reads data from the camera 237 at a higher bitrate when thevalues are indicative of the EMV being below a threshold compared to alower bitrate at which data is read from the camera 237 when the valuesare indicative of the EMV being above the threshold. Additionally oralternatively, the computer 240 may calculate pupil features 246, basedon the images 239, at a higher rate when the values are indicative ofthe EMV being below a threshold compared to when the values areindicative of the EMV being above the threshold.

EMV can be calculated by the computer 240 based on the change to eyepositions over time. For example, at a certain time, the velocity may becalculated based on differences between two or more measurements of eyepositions, conducted over a short period (such as successive eyeposition calculations performed based on the measurements 236 of thereflections). To calculate the eye positions and/or eye movements, thecomputer 240 may utilize various techniques known in the art, such asthe techniques described in Rigas 2017 and/or other references involvingPSOG, which are mentioned herein. Eye positions and eye velocities aretypically described in terms of angular movement. Thus, the EMV can beexpressed in some embodiments in term of angular velocity (i.e.,degrees/second). Alternatively, in some embodiments, EMV may beexpressed in absolute distances in which the eye moves. For example, inthese embodiments, the EMV and the threshold may be expressed in unitsof centimeters/second.

In one example, the threshold may correspond to an EMV of 40°/second. Inanother example, the threshold may correspond to an EMV of 30°/second.In yet another example, the threshold may correspond to an EMV of100°/second. In still other examples, the threshold may have a valuethat corresponds to an EMV that is a certain value that falls within therange of 20°/second to 150°/second. Herein, “bitrate” refers to thevolume of data (e.g., expressed in bits or bytes) that is read in agiven period of time. For example, a bitrate as used herein may beexpressed in terms of bytes/second.

In some examples, the values indicative of EMV, which are calculatedbased on the measurements 236 of the reflections, may include values ofangular velocity of the eye (e.g., expressed as a degrees/second of eyemovement). In other examples, the values indicative of EMV, may includevalues amplitudes of eye movements expressing the angular distance aneye traveled during a certain window of time.

In some embodiments, the rate at which the values indicative of EMV arecalculated is higher than the rate at which the images 239 are captured.For example, the images 239 may be captured at a rate that is below 100Hz (e.g., 50 or 60 Hz), while the values are calculated at a rate thatis significantly higher, such as 500 Hz, 1000 Hz, or more.

Reading data from the camera 237 at a higher bitrate, such as when thevalues are indicative of the EMV being below the threshold, compared toa lower bitrate, such as the bitrate at which data is read from thecamera 237 when the values are indicative of the EMV being above thethreshold, may be done in different ways in different embodiments.

In one embodiment, the computer 240 does not read image data from thecamera 237 when the values calculated based on the measurements 236 ofthe reflections are indicative of the EMV being above the threshold.Optionally, the bitrate at which data is read from the camera 237 whenthe values are indicative of the EMV being above the threshold is lessthan 10% of the bitrate at which data is read from the camera 237 whenthe values are indicative of the EMV being below the threshold.Optionally, the bitrate at which data is read from the camera 237 whenthe values are indicative of the EMV being above the threshold is zero.

In another embodiment, the computer 240 reads image data at a higherfrequency when the values calculated based on the measurements 236 ofthe reflections are indicative of the EMV being below the thresholdcompared to when said values are indicative of the EMV being above thethreshold. Optionally, the resolution of the images read from the camera237 at these different times is the same. In one example, the computer240 reads first images, from among the images 239, at a frequency thatis at least 50 Hz when the values are indicative of the EMV being belowthe threshold, and the computer 240 reads second images, from among theimages 239, at a frequency that is below 50 Hz when the values areindicative of the EMV being above the threshold. In another example, thecomputer 240 reads images, from among the images 239, at a frequencythat is below 10 Hz when the values are indicative of the EMV beingabove the threshold.

In yet another embodiment, the computer 240 reads images with a higherresolution when the values calculated bases on the measurements 236 ofthe reflections are indicative of the EMV being below the thresholdcompared to when said values are indicative of the EMV being above thethreshold. Optionally, the frequency at images read from the camera 237at these different times is the same.

Calculations of the values indicative of the EMV may be done at variousrates in different embodiments. In some embodiments, calculations ofthese values are performed at a frequency at which measurements 236 ofthe reflections are taken by the PSOG 235, such as 500 Hz, 1000 Hz ormore. In other embodiments, the values indicative of the EMV may becalculated at a lower frequency than the frequency at which themeasurements 236 of the reflections are taken. For example, the valuesmay be calculated at frequency of 100 Hz or lower. Thus, decisions madebased on values indicative of the EMV, such as a rate at which to readdata from the camera 237 and/or a rate at which to calculate the pupilfeatures 246 may be done at various frequencies. For example, thesettings 238 of the camera may be updated at different rates orasynchronously (e.g., as the indication of whether the EMV is above orbelow the threshold change).

Different eye tracking parameters may bring different values indifferent scenarios/experiments. These differences can help the systemto save power by using the EMV to set the computer 240 to read the datafrom the camera 237 at different bitrates during eye fixation, smoothpursuits, and saccades. For example, when a researcher is interested inspatial attention, eye tracking data during a smooth pursuit may be morevaluable than eye tracking data during fixation, and thus the system cansave power by reading the data from the camera at a higher bitrateduring smooth pursuits versus the bitrate at which data is read duringeye fixations. And when the researcher is more interested in generatinga heat map representing visual attention, then eye tracking dataobtained during a fixation may be more valuable than eye tracking dataobtained during a smooth pursuit, and the system may read the data fromthe camera 237 at a higher bitrate during fixations versus the bitrateat which data is read during smooth pursuits.

The computer 240 may utilize, in some embodiments, the values indicativeof the EMV, to detect eye fixations, smooth pursuit eye movements,and/or saccades, and to read the data from the camera 237 at differentbitrates during the eye fixations, the smooth pursuit, and the saccades.Optionally, average bitrates at which the data is read from the camera237 during the eye fixations and the smooth pursuit eye movements are atleast three times greater than an average bitrate at which the data isread from the camera 237 during saccades.

In one embodiment, the computer 240 is configured to set timing of thecamera 237 to capture above 80% of the images 239 when the valuescalculated based on measurements 236 of the reflections are indicativeof the EMV being below the threshold. In one example, having the EMV beabove the threshold is indicative of saccades, and on average, imagescaptured when there are no saccades are sharper than images capturedduring saccades.

When used for eye tracking, a PSOG-based system will typically consumesignificantly less power than a VOG-based system, if eye parameters arecalculated by both systems at the same rate. Therefore, a combination ofboth PSOG and VOG can enable utilization of the typically more accurate(but power hungry) VOG at times that are likely to yield more usefuland/or accurate eye parameters. Thus, identifying when the EMV is belowthe threshold (using the PSOG), can be used to trigger the use of VOG tocalculate eye parameters during eye fixations, while avoiding the use ofVOG during times of saccades that are likely to yield less accurate oruseful eye parameters.

In one example, the PSOG 235 consumes below 1 mW while VOG (using thecamera 237) consumes above 100 mW for eye tracking at the same rate.Because the higher the bitrate read from the camera 237 the more poweris consumed for eye tracking that is based (at least in part) on imagestaken by the camera 237, using the PSOG 235 to trigger reading thecamera 237 when the values are indicative of the EMV being below thethreshold should reduce the power consumption of the system. Examples ofvalues for the threshold may be around 40°/second (i.e., angular speedof the eye of 40° per second), to cover normal smooth pursuits thethreshold may be up to 30°/second, and the threshold may be around100°/second to cover fast smooth pursuits combined with catch-upsaccades.

PSOG, as the term is used herein, may involve utilization of one or morelight sources and/or one or more detectors, such as discretephotosensors, that detect reflections of the light emitted from the oneor more light sources (and possibly external light sources too).

FIG. 2A illustrates smartglasses with PSOG that utilizes multiple lightsources and multiple detectors. Coupled to the frame of thesmartglasses, around each of the lenses, are multiple photosensors 228and multiple light sources (emitters 229), which are interleaved betweenthe photosensors 228. It is to be noted that the emitters andphotosensors are present on the frame around both lenses, but for thesake of clarity the photosensors 228 are designated on the left side andthe emitters 229 are designated on the right side.

In one embodiment, the PSOG 235 includes: at least two light sourcesconfigured to emit the light, and at least three discrete photosensorsconfigured to measure the reflections. Optionally, the discretephotosensors are spread over more than 2 cm.

In another embodiment, the PSOG 235 comprises at least two LightEmitting Diodes (LEDs) having a bi-directional characteristic with theability to emit the light and to measure the reflections. Optionally,each of the at least two LEDs is sensitive to wavelengths equal to orshorter than the predominant wavelength it emits. Optionally, each ofthe at least two LEDs provides illumination when a forward voltage isapplied to its electrical terminals, and acts asphotodetector/photodiode for example by the following three steps: (i)apply a reverse voltage pulse for a short duration, (ii) discharge theLED's capacitance immediately afterwards, and (iii) measure the voltageacross the LED to determine how much discharge of capacitance took placeafter a certain time. This technique is well known in the art and isfurther explained in publications such as (A) Akşit, Kaan, Jan Kautz,and David Luebke “Gaze-Sensing LEDs for Head Mounted Displays” arXivpreprint arXiv:2003.08499 (2020), and (B) Dietz, Paul, WilliamYerazunis, and Darren Leigh “Very low-cost sensing and communicationusing bidirectional LEDs” International Conference on UbiquitousComputing, Springer, Berlin, Heidelberg, 2003.

In some embodiments, the camera 237 utilizes a sensor that has more than100 pixels. In these embodiments, the camera 237 may also have a lens,and the sensor plane of the camera 237 may be tilted by more than 2°relative to the lens plane of the camera 237, according to theScheimpflug principle in order to capture sharper images.

FIG. 2B illustrates an embodiment of an eye tracking system onsmartglasses that tracks both eyes, which utilizes multiple lightsources and detectors to track each eye. The illustrated system includesthe smartglasses 230 that have PSOG and VOG that may be used together totrack movements of both eyes. Tracking of the left eye is done utilizinga PSOG that includes multiple light sources (emitters 231 a and 231 b inthe figure) as well as multiple detectors (discrete photosensors 232 a,232 b, and 232 c). Additionally, video camera 233 a may be utilized tocapture images of the left eye, which can be used to determine positionsand/or movements of the left eye. In a similar fashion, tracking theright eye is done in this embodiment utilizing another PSOG thatincludes additional light sources (emitters 231 c and 231 d in thefigure) as well as additional multiple detectors (discrete photosensors232 d, 232 e, and 232 f) and an additional video camera 233 b that maybe utilized to capture images of the right eye.

Knowing positions and/or movements of both eyes can be utilized toestimate the gaze of the user. In one embodiment, the eye trackingsystem includes an another PSOG (e.g., as illustrated in FIG. 2B) Inthis embodiment, the computer 240 calculates a gaze direction based onthe measurements 236 of the reflections, measured by the PSOG 235, andadditional measurements of reflections (of the other eye) taken by theother PS OG. For example, the computer 240 may utilize techniques knownin the art to determine the user's gaze direction based on the eyepositions of each of the eyes as determined based on measurements of thePSOG 235 and the additional measurements of reflections taken by theother PSOG. For example, the gaze may be determined using one or more ofthe computational techniques discussed in the references mentioned abovethat disclose a standalone PSOG.

Light sources may be utilized in different ways in order to conservepower and/or improve quality of images taken by the eye tracking system.In one embodiment, the computer 240 is configured to command the PSOG235 to emit light with a higher intensity while the images 239 are beingtaken, compared to the average intensity of light emitted by the PSOG235 while measuring reflections at times at which the images 239 are notbeing taken. Added illumination by emitters of the PSOG 235 may assistin making the images 239 clearer, compared to images taken while theemitters of PSOG 235 do not operate. In another embodiment, in which thePSOG 235 comprises multiple light sources that emit the light, whilemost of the measurements 236 of the reflections are taken, the PSOG 235emits light from a single light source at a time. In this embodiment,for most of the images 239 that are captured, the PSOG 235 emits lightfrom multiple light sources during the exposure time of the camera 237(in which the images 239 are captured).

Utilizing the PSOG 235 to determine when and/or to what extent toutilize the camera 237 for eye tracking can be useful when the camera237 is battery-operated, such as when it is embedded in a wearabledevice (e.g., smartglasses) or a mobile device (e.g., a smartphone).

In one embodiment, the camera 237 and the computer 240 are head-mountedand the PSOG 235 is more power-efficient than the camera 237 (meaningthat if the PSOG 235 and the camera 237 operate taking their respectivemeasurements for the same amount of time, the PSOG 235 uses less powerthan the camera 237). Additionally, in this embodiment, the threshold towhich the EMV is compared is 40°/second. In this embodiment, the eyetracking system illustrated in FIG. 1 may save power as a result ofreducing the bitrate at which the data is read from the camera 237 whenthe values calculated based on the measurements 236 of the reflectionsare indicative of the EMV exceeding the threshold of 40°/second.

In another embodiment, the camera 237 may belong to a non-head-mountedbattery-operated mobile device, reference to the computer 240 mayinvolve, in this embodiment, at least a first component that is ahead-mounted computer and a second component that is a non-head-mountedcomputer, which are configured to communicate over a wirelesscommunication channel. In this embodiment, reducing the bitrate at whichthe data is read from the camera 237 when the values are indicative ofthe EMV exceeding the threshold may reduce power consumption of thenon-head-mounted device. Such a scenario is illustrated in FIG. 3 ,where the camera 237 is embedded in a non-head-mounted battery-operatedmobile device (a smartphone 244), while the PSOG 235 is part of ahead-mounted device (smartglasses 243). Thus, in this example referencesto “the computer 240” may refer to a head-mounted computer 241 a (e.g.,a processor of the smartglasses 243) and/or a non-head-mounted computer241 b (e.g., a processor in the smartphone 244).

Given the importance of eye tracking for many applications, many systemsinclude both PSOG and VOG components (e.g., the PSOG 235 and the camera237, respectively), which are often used in a manner that is intended toimprove performance tradeoffs. For example, the more power-demanding VOGmay be utilized sporadically to correct and/or complement the lessaccurate but less power-demanding PSOG. One way in which PSOG and VOGmay be utilized together is in the training and/or calibration of modelsfor detecting eye movement and/or eye positions based on PSOG (e.g.,determining eye positions based on the measurements 236 of thereflections). In some embodiments, data obtained from VOG (e.g., byanalyzing the images taken by the camera 237) is utilized as a “groundtruth” indicating eye movements and/or eye positions at the time theimages were taken. This data can then be utilized to train a model(e.g., set parameters of the model) for power-efficient eye trackingthat maps measurements of reflections obtained with PSOG to eyemovements and/or eye positions.

In one embodiment, in order to train such a model, the computer 240receives images captured by the camera 237 and calculates eye positionsand/or eye movements based on analysis of the images (referred to hereinas “image-based positions”). The image-based positions serve as labels(the ground truth) for training the model. Together with correspondingmeasurements of reflections measured at corresponding times with thePSOG 235, a training set of samples is generated. This training set isprovided to a machine learning training algorithm and is used to trainand/or update the model. In one example, the model may be a regressionmodel. In another example, the model may include parameters of aconvolution neural network. Additional details about how such models maybe trained are provided in Rigas 2017.

In some embodiments, calibration approaches known in the art fordetermining eye positions based on PSOG may be adapted to someembodiments of the system illustrated in FIG. 1 . For example, insteadof obtaining ground truth positions by providing the user with cuesindicating where to look, the approach described in Topal, et al. “Alow-computational approach on gaze estimation with eye touch system.”IEEE transactions on cybernetics 44.2 (2013): 228-239, can be adapted touse ground truth positions obtained determined based on images from thecamera 237. In another example, eye positions can be determined based onimages obtained by the camera 237 in a similar way to how ground truthpositions are obtained with the IR camera in Li, et al. “Battery-freeeye tracker on glasses.” Proceedings of the 24th Annual InternationalConference on Mobile Computing and Networking. 2018.

The camera 237 may be operated at different times and/or at differentfrequencies in order to capture images that are utilized to calculatethe image-based positions. For example, the camera 237 may be operatedmore when it is determined that a model used to calculate eye positionsand/or eye movements based on PSOG may be inaccurate and/or may requirecalibration.

Each time a person wears the smartglasses there may be slight variationsin the way the smartglasses fit on the face. These variations mayinvolve slight sensor shifts, which can change the nature of thecorrelations between reflections measured with PSOG when the eye is atcertain positions. In order to account for these variations, models usedto calculate eye positions and/or eye movements based on PSOG may needto be updated and/or recalibrated. To this end, in some embodiments,when it is determined that the user put on the smartglasses 230, thecamera 237 is operated in order to capture images that may be utilizedas the ground-truth image-based positions.

In order to maintain accurate eye tracking results from the PSOG, insome embodiments, the camera 237 may be periodically be operated tocapture images that are utilized as a ground truth according to whichthe accuracy of eye positions determined based on measurements of thePSOG 235 is evaluated. For example, camera 237 may be operated every fewseconds and/or after certain events (e.g., after intense head movementsare detected). Optionally, if the accuracy of the eye positions that aredetermined based the PSOG 235 falls below a certain threshold, then thecamera 237 is operated at a higher frequency in order to provideadditional images to serve for calibrating and/or retraining the modelused to calculate the eye positions based on the measurements from thePSOG 235.

Extensive eye movements can be indicative of unrest of the user. In oneembodiment, the computer 240 sums durations during which the EMVexceeded the threshold, and responsive to the sum reaching a secondthreshold, commands a user interface to present a calming message to theuser (e.g., by displaying text with the message to the user or playingto the message via headphones worn by the user). In one example, thecalming message may include an instruction to close the eyes and relax,or be a message that comes from a digital therapist application that isrun in order to help the user to relax.

In one embodiment, the computer 240 utilizes at least one of thediscrete photosensors utilized by the PSOG 235 to detect a flickeringtiming of ambient light, and set the camera 237 to capture the images239 as a function of the flickering timing. For example, when capturingimages during more light flickering periods improves the quality of thecaptured images, the computer may operate the camera according to theflickering timings in order to capture more images during the flickeringperiods.

In one embodiment, the PSOG 235 is mounted to a smartglasses framecomprising progressive ophthalmic lenses, and the computer is furtherconfigured to: receive an indication that the user is characterized asdiabetic, detect, based on the values indicative of the EMV, shorteningof averaged fixation periods, and command a user interface to suggest tothe user to check his/her blood sugar level. The shortening of theaveraged fixation periods may be indicative of diabetic retinopathy.

In another embodiment, the PSOG 235 is mounted to a smartglasses framecomprising progressive ophthalmic lenses, and the computer 240 isfurther configured to: receive an indication that the user suffers fromdry eye syndrome, detect, based on the values indicative of the EMV,shortening of averaged fixation periods, and command a user interface tosuggest to the user to treat the dry eye condition. The shortening ofthe averaged fixation periods may be indicative of the eye being toodry.

The following method may be used by systems modeled according to FIG. 1. The steps described below may be performed by running a computerprogram having instructions for implementing the method. Optionally, theinstructions may be stored on a computer-readable medium, which mayoptionally be a non-transitory computer-readable medium. In response toexecution by a system including a processor and memory, the instructionscause the system to perform the following steps:

In Step 1, emitting light and taking measurements of reflections of thelight from an eye of a user by a photosensor-oculography device (PSOG).

In Step 2, capturing images of the eye measured in Step 1 by a camera.For example, the camera 237 may be utilized to capture such images.

In Step 3, calculating values indicative of eye movement velocity (EMV)based on the measurements of the reflections.

And in Step 4, reading data from the camera at a higher bitrate when thevalues are indicative of the EMV being below a threshold compared towhen the values are indicative of the EMV being above the threshold.

In one embodiment, the method may optionally include the followingsteps: detecting, based on the values indicative of the EMV, eyefixations, smooth pursuit eye movements, and saccades, and reading thedata from the camera at different bitrates during the eye fixations, thesmooth pursuit, and the saccades. Optionally, average bitrates at whichthe data is read from the camera during the eye fixations and the smoothpursuit eye movements are at least three times greater than an averagebitrate at which the data is read from the camera during saccades.

In another embodiment, the method may optionally include the followingsteps: calculating eye positions and/or eye movements based on analysisof the images (image-based positions), generating labels based on theimage-based positions, generating feature values based on themeasurements of the reflections, and providing the labels and featurevalues to train a machine learning-based model for detecting eyepositions and/or eye movements based on additional measurements ofreflections.

Some embodiments described herein involve calculation of the pupilfeatures 246 based on the images 239 taken by the camera 237. Pupilfeatures of an eye that appears, at least in part, in images may includeone or more of the following types of values: a pupil center location, apupil contour, a pupil diameter, glint location, a glint-pupil vector, aHaar-like feature, a result of an ellipse fitting approach, and a resultof pupil edge filtering. The aforementioned pupil features are wellknown in the art, and discussed in the reference Fuhl, Wolfgang, et al.“Pupil detection for head-mounted eye tracking in the wild: anevaluation of the state of the art” Machine Vision and Applications 27.8(2016): 1275-1288, which is incorporated herein by reference.

One advantageous aspect of some of the embodiments described herein isthat they enable calculation of the pupil features in a power-efficientway by selectively choosing when to calculate the pupil features at ahigher rate. For example, the pupil features may be calculated when theyare more likely to be informative and/or accurate (e.g., at times inwhich the eye is relatively stationary). Optionally, determining whenpupil features are more likely to be accurate and/or informative is donebased on the values indicative of eye movement velocity (EMV), which arecalculated based on the measurements 236 of the reflections.

Thus, in some embodiments, the computer 240 may be configured to obtainthe images 239, and/or calculate the pupil features 246, at a higherrate during periods of eye fixations compared to periods of saccades.This can be beneficial because, on average, images captured when thereare no saccades are sharper than images captured during saccades (whichmay be blurry and/or include artifacts due to the high velocity theeye's movements).

In one embodiment, the computer 240 calculates the pupil features 246during eye fixations at a rate that is at least three times higher thana rate at which the pupil features 246 are calculated during saccades.

In another embodiment, the computer 240 sets timing of the camera 237 tocapture above 80% of the images when the values are indicative of theEMV being below the threshold. For example, the computer 240 maydecrease the frequency at which the camera 237 captures images, orrefrain from capturing images altogether, when the values calculatedbased on the measurements 236 of the reflections are indicative of theEMV being above the threshold. Optionally, having the EMV be above thethreshold is indicative of saccades (i.e., that the eye is performingsaccadic movements).

In yet another embodiment, the computer 240 identifies saccades based onthe values calculated based on the measurements 236 of the reflections.The computer 240 utilizes identifications of saccades for timing thecamera to capture above 80% of the images when there are no saccades.

Identifying whether the eye is in the midst of saccadic movements may bedone in different ways. In some embodiments, if the values calculatedbased on the measurements 236 of the reflections are indicative of theEMV reaching a threshold, this may be interpreted by the computer 240 asan occurrence of a saccade (i.e., the eye is performing a saccadicmovement). In other embodiments, the computer 240 may utilize a machinelearning-based approach to identify a saccade. For example, the computer240 may generate feature values based on the measurements 236 of thereflections, and utilize a model to calculate, based on the featurevalues a certain value indicative of whether the user's eye is in asaccade movement. Optionally, if the certain value indicates that theprobability that the eye is in a saccade is above a certain threshold,then then the computer 240 reduces the frequency at which the camera 237captures images or has the camera 237 refrain from capturing images fora certain period (e.g., a few tenths of a second) and/or until thecertain value, when calculated based on later measurements ofreflections, indicates that the eye is no longer in saccade movement.Optionally, the feature values may include values indicative of extentsof reflections at different times (e.g., raw values measured by thePSOG), and/or various values that may be functions of these values, suchas volume local binary patterns (VLBP), cuboids, and/or opticalstrain-based features. Optionally, the model utilized to calculate thecertain value is trained based on data comprising previous measurementsof reflections of the eye of the user measured by the PSOG 235 andprevious images of the eye of the user captured with the camera 237. Totrain the model, feature values were generated based on the previousmeasurements, and labels indicative of whether the user's eye was in asaccade movement were generated based on the previous images usingtechniques known in the art for identifying saccade movements. In oneexample, the model may include parameters of a regression model (e.g., alinear regression model or a logistic regression model). In anotherexample, the model may include parameters of a convolution neuralnetwork.

During periods of saccades there is often less visual processingperformed by the brain. Thus, certain pupil features, associated withvisual processing, can be less informative, and/or provide less utility,during periods of saccades. Therefore, in some embodiments, the computer240 identifies saccades based on the values calculated based on themeasurements 236 of the reflections, and calculates the pupil diameter,based on the images 239, at a higher rate during periods withoutsaccades compared to periods of saccadic movements.

In some embodiments, due to optical design of the camera 237 and/or thelocation of the camera 237 relative to the eye, images of the pupil aresharper when a gaze direction of the user is in a first directioncompared to when the gaze direction of the user is in a seconddirection. Optionally, the first and second gaze directions differ by atleast 20°. Optionally, the first and second gaze directions differ by atleast 45°. In one embodiment, the computer 240 calculates an eyeposition based on the measurements 236 of the reflections. Optionally,the eye position is indicative of a direction of the user's gaze, whichis indicative of whether the gaze is closer to the first direction thanit is to the second direction. Knowing the gaze direction can enablesaving power. For example, the computer 240 can calculate the pupilfeatures more frequently based on images taken while the gaze directionis in the first direction, compared to images taken while the gazedirection is in the second direction.

In one embodiment, the computer 240 calculates the eye position based onthe measurements 236 of the reflections at a rate that is at least tentimes higher than a rate at which the pupil features 246 are calculatedbased on the images 239. This difference in the rate of calculating theeye position based on the measurements 236 of the reflections versus therate of the calculating the pupil features 246 based on the images 239can further reduce the power consumption of the system.

Utilizing the PSOG 235 to determine when, and/or to what extent, tocalculate pupil features can be advantageous when the camera 237 and/orthe computer 240 are battery-operated, such as when these components areembedded in a wearable device (e.g., smartglasses) or a mobile device(e.g., a smartphone).

In one embodiment, the camera 237 belongs to a non-head-mountedbattery-operated mobile device, and “the computer 240” comprises ahead-mounted computer and a non-head-mounted computer, which areconfigured to communicate over a wireless communication channel.Reducing the rate at which the pupil features 246 are calculated, whenthe values are indicative of the EMV exceeding the threshold, can savepower expenditure of the non-head-mounted device. For example, thehead-mounted computer may be embedded in smartglasses, thenon-head-mounted computer may be embedded in a computer smartphone, thecamera 237 may be the front-facing camera of the smartphone, thesmartphone may run an eye tracker, and the wireless communicationchannel may be Bluetooth Low Energy. In this embodiment, the thresholdto which the EMV is compared may be selected to exclude saccades. Forexample, reaching the threshold indicates that a probability that theeye is in a saccadic movement is at least a certain probability, and thecertain probability is greater than 50%. FIG. 3 illustrates such anembodiment. This example reduces the power consumption of running theeye tracker on the smartphone 244 by making it unnecessary for thesmartphone's eye tracker to attempt to calculate pupil features duringsaccades. One example of a smartphone eye tracker, which can be adaptedto save power by refraining from analyzing images taken during saccades,is the multi-layer feed-forward convolutional neural network eye trackerdisclosed in the reference Valliappan, Nachiappan, et al. “Acceleratingeye movement research via accurate and affordable smartphone eyetracking” Nature communications (2020).

Variations due to flickering ambient light can affect the quality ofpupil features calculated based on the images 239. Thus, in someembodiments, the computer 240 is configured to utilize at least one ofthe discrete photosensors utilized by the PSOG 235 to detect aflickering timing of ambient light, and adjust periods during which thelight is emitted by the PSOG 235 according to the flickering timing ofthe ambient light. Optionally, timing the periods during which the lightis emitted by the PSOG 235 according to the flickering timing of theambient light improves the signal-to-noise ratio of the measuredreflections, which improves the computer's ability to calculate theeye-related parameters based on measurements of reflections measured bythe PSOG 235. Additionally or alternatively, timing the periods at whichthe PSOG 235 emits light can help to provide more consistentillumination to images captured during those times, which can assist inimproving the quality of pupil features calculated based on the imagescaptured at those times.

In one example, the periods during which the light is emitted are timedby the computer 240 to coincide with periods between the flickering ofthe ambient light. In another example, the periods during which thelight is emitted are timed by the computer 240 to coincide with periodsof low intensity ambient light from a flickering ambient light source.

The following method may be used by systems modeled according to FIG. 1. The steps described below may be performed by running a computerprogram having instructions for implementing the method. Optionally, theinstructions may be stored on a computer-readable medium, which mayoptionally be a non-transitory computer-readable medium. In response toexecution by a system including a processor and memory, the instructionscause the system to perform the following steps:

In Step 1, emitting light and taking measurements of reflections of thelight from an eye of a user by a photosensor-oculography device (e.g.,the PSOG 235).

In Step 2, capturing images of the eye measured in Step 1 by a camera.For example, the camera 237 may be utilized to capture such images.

In Step 3, calculating values indicative of eye movement velocity (EMV)based on the measurements of the reflections.

And in Step 4, calculating pupil features, based on the images, at ahigher rate when the values calculated in Step 3 are indicative of theEMV being below a threshold compared to when those values are indicativeof the EMV being above the threshold.

In one embodiment, having EMV below a first threshold is indicative ofan eye fixation and having the EMV above a second threshold isindicative of a likely saccade. In this embodiment, the pupil featuresare calculated during eye fixations (i.e., when the values calculated inStep 3 are below the first threshold) at a rate that is at least threetimes higher than a rate at which the pupil features are calculatedduring saccades (i.e., when the values are above the second threshold).

The values calculated in Step 3 may be utilized in various ways todetermine when and/or to what extent images are captured by the camera.Thus, fewer images may be captured when pupil features calculated basedon the images are likely to be less accurate and/or less informative.

In one embodiment, the method described above also includes a step thatinvolves utilizing the values indicative of the EMV in order to operatethe camera at a higher frame rate when the values are indicative of theEMV being below the threshold compared to when the values are indicativeof the EMV exceeding the threshold.

In one embodiment, the method optionally includes a step of settingtiming of the camera to capture above 80% of the images when the valuesare indicative of the EMV being below the threshold. The fact that theEMV is above the threshold may be indicative of saccades, and onaverage, images captured when there are no saccades are sharper thanimages captured during saccades.

In another embodiment, the method optionally includes a step ofidentifying saccades based on the values, and utilizing identificationsof the saccades for timing the camera to capture above 80% of the imageswhen there are no saccades.

In yet another embodiment, the method optionally includes a step ofidentifying saccades based on the values, and calculating the pupildiameter, based on the images, at a higher rate during periods withoutsaccades compared to periods of saccadic movements. It is noted thathere is less visual processing during saccades and therefore it is lessvaluable to measure changes in pupil diameter during saccades.

FIG. 4 illustrates an embodiment of an eye tracking system that utilizesan event camera. In some embodiments, the system includes at least aphotosensor-oculography device (PSOG 262), which is configured to emitlight and to take measurements 263 of the reflections of the light froman eye of a user, an event camera 264 that is configured to capture theeye and to provide events 265, and a computer 268 that calculatespositions of the eye (eye positions 269) based on the events 265 and themeasurements 263. Optionally, one or more components of some embodimentsof the system illustrated in FIG. 4 , such as the PSOG 262, the eventcamera 264, and/or the computer 268 may be head-mounted and/or coupledto (and/or embedded) in frames of smartglasses 260 that are configuredto be worn on the user's head.

It is to be noted that “capturing the eye” by the event camera 264refers to taking measurements resulting from capturing (detecting)photons reflected from the eye, these can result in the events 265and/or frame images 266 (which are discussed below).

Due to the nature of how the signals are acquired, the measurements 263of the reflections and the events 265 will often be received indifferent manners. Namely, the events 265 are typically detectedasynchronously, while the measurements 263 may be obtained in asynchronous manner (e.g., during certain periods, the PSOG 262 may beoperated at a fixed frequency at which it emits lights and measuresreflections from the eye). Additionally, due to the nature of theoperation of event cameras, which enable a quick reading of singlepixels, in some embodiments, the rate at which the events 265 arereceived can be higher than the rate at which the measurements 263 areacquired. Utilizing both the measurements 263 and the events 265 tocalculate the eye positions 269 can leverage the differentcharacteristics of these signals to improve performance of an eyetracker system that utilizes the PSOG 262 and the event camera 264, interms of accuracy, frequency, and/or reduced power usage. The followingis a description of some of the ways in which the measurements 263 ofthe reflections and the events 265 may be utilized by the computer 268in order to calculate the eye positions 269, which may confer some ofthe aforementioned advantages.

One approach that may be utilized by the computer 268, relies in thefact that in some embodiments, the event camera 264 is able to generateevents at a much higher rate compared to the rate at which the PSOG 262operates. In these embodiments, the computer 268 may use themeasurements 263 to calculate eye positions (referred to herein as “maineye positions”) and use the faster occurring events 265 to update theeye positions at times that fall between consecutive main eye positionscalculated based on the measurements 263 (these updated eye positions,which are calculated based on the events 265 are referred to herein as“intermediate eye positions”). Thus, the eye positions 269 may include astream of positions that combines both the main eye positions,calculated from the measurements 263 obtained from the PSOG 262, and theintermediate eye positions, calculated by updating the main eyepositions according to events detected by the event camera 264, whichwere detected in between the times corresponding to the main eyepositions. Optionally, the computer 268 calculates, based on themeasurements 263 a stream of main eye positions, and calculates based onthe events 265 intermediate streams of eye positions which are locatedbetween at least some of the main eye positions. Optionally, on average,time that elapses between calculation of consecutive eye positions inthe intermediate streams is much shorter than average time that elapsesbetween calculation of consecutive eye positions from among the main eyepositions.

The main eye positions calculated based on the measurements 263 mayinclude a parametric representation of the eye. One example of aparametric representation is described in the reference Angelopoulos, etal., “Event Based, Near Eye Gaze Tracking Beyond 10,000 Hz” arXivpreprint arXiv:2004.03577 (2020). Angelopoulos, et al. describe howimage data can be utilized to represent an eye position using a modelwith 13 parameters describing properties that include parameters of anellipse representing the pupil, a parabola representing the eyelashes,and a circle representing the glint (a reflection of an IR light sourceoff of the user's eyeball). The parameters are fit based on dataobtained from images, and updated incrementally based on events. In asimilar fashion, the computer 268 may fit parameters of a model of theeye based on the measurements 263 to obtain the main eye positions,e.g., using one or more of the standalone PSOG eye tracking approachesmentioned herein. These main eye positions can then be updated based onthe events 265 in a similar fashion to the updating of eye positionsdescribed in Angelopoulos, et al.

Thus, the resulting calculations performed by the computer 268 can beviewed, in some embodiments, as a fusion algorithm that combines betweenthe measurements 263 and the events 265. In one embodiment, this fusionalgorithm includes the following steps: In step 1, set an initial eyeposition based on a subset of the measurements 263 taken a time t. Instep 2, receive a subset of the events 265 that were detected after thetime t and calculate updated eye positions by adding shifts obtainedfrom these events to the initial eye position, until new measurements ofreflections are received at time t+Δ. And in step 3, go back to step 1to set the initial eye position based on the measurements of reflectionsreceived at t+Δ.

It is to be noted that due to the speed at which events are detected, aneye tracker that utilizes a fusion approach, as described above, mayachieve a very high tracking frequency. For example, in someembodiments, the PSOG 262 may operate at a frequency of 200 Hz or even1000 Hz. However, with the combination of updating eye positions usingthe events 265, the tracker is capable of providing eye positions at afrequency of 10,000 Hz or more.

Another approach that may be utilized by the computer 268 in thecalculation of the eye positions 269 involves selection of which datafrom the different devices to use (from among the measurements 263 andthe events 265) based on which is appropriate at a given time, accordingto a detected behavior of the eye. This may save power and/or increaseaccuracy of eye tracking.

In one embodiment, the computer 268 operates the event camera 264 in anormal mode while eye movement velocity (EMV) is above a threshold, andoperates the event camera 264 in a low-power mode during fixations(which, for example, may cause it to detect fewer events) or refrainsfrom processing the events 265 during the fixations. Optionally, thecomputer 268 calculates the eye positions 269 based on the measurements263 during the fixations. Optionally, the computer 268 calculates theEMV based on the measurements 263, e.g., as discussed above withreference to calculations of the EMV by the computer 240 based on themeasurements 236 of the reflections. It is to be noted that thethreshold mentioned with respect to the EMV that is calculated based onthe measurements 236 need not be the same threshold mentioned hereinthat relates to the EMV that is calculated based on the measurements263.

In one embodiment, the computer 268 operates the PSOG 262 in a normalmode during fixations, and detects when the EMV reaches the thresholdbased on the measurements 263. Optionally, the computer 268 operates thePSOG 262 in a low-power mode when the EMV is above the threshold, andcalculates the eye positions 269 based on the events 265 while the EMVis above the threshold.

Herein, a low-power mode for PSOG is a mode in which, over a certainperiod of time, the PSOG consumes less power than it would were it tooperate in a normal mode. Optionally, power consumption of PSOG in thelow-power mode is less than 50% of the power it consumes when operatingin the normal mode. In some embodiments, achieving the lower powerconsumption of the low-power mode may involve one or more of thefollowing: refraining from emitting and/or measuring reflections by thePSOG, reducing the frequency at which measurements are taken with thePSOG, reducing the extent of illumination by emitters of the PSOG,reducing the number of emitters of the PSOG that are operated, and/orreducing the number of discrete photosensors of the PSOG that are read.

In some embodiments, the total power consumed by the event camera 264 ishigher than the total power consumed by the PSOG 262 (when operated in anormal operation mode for the same duration of time). Therefore, an eyetracker system that is based on these two components can save power bycalculating the eye positions 269 based on measurements of the PSOG 262(possibly without using events detected by the event camera 264) whenthe EMV is below the threshold, and calculating eye positions based onevents 265 measured by the event camera 264 (possibly without using themeasurements taken by PSOG 262) when the EMV is above the threshold.

The threshold for the EMV may be selected according to one or more ofthe following considerations: (i) performances, accuracy, and/or delayof eye positions calculated by the PSOG 262 as a function of the EMV,(ii) performances, accuracy, and/or delay of obtaining eye positionscalculated by the event camera 264 as a function of the EMV, and (iii)power consumption of the PSOG 262 vs power consumption of the eventcamera 264, as a function of the velocity of eye movements that are tobe tracked. For example, the threshold for the EMV may be 100°/secondfor a system that operates the PSOG 262 at a relatively low frequencythat is not designed to measure saccades. Alternatively, the thresholdfor the EMV may be 400°/second for a system that operates the PSOG 262at a higher frequency that enables it to measure normal saccades.

Operating the PSOG 262 may introduce, in some scenarios, artifact eventsthat may be detected by the event camera 264, which are due to the lightemitted by the PSOG 262, and not, for example, because of movement ofthe eye. To reduce the effect of such artifacts and/or overcome theeffects altogether, in some embodiments, the light emitted by the PSOG262 is in a first spectrum band, the event camera 264 is configured tocapture the eye at a second spectrum band, and the first and secondspectrum bands are disjoint. In other embodiments, the PSOG 262 may emitlight in a certain spectrum band, and the event camera 264 may beequipped with a filter that attenuates most of the light detected in thecertain spectrum band. For example, the filter may attenuate more than95%, more than 98%, or more than 99.9% of the light in the certainspectrum band. Thus, the PSOG 262 and the event camera 264 may beconsidered to operate in spectrum bands that are essentially disjoint.Having the PSOG 262 and the event camera 264 operate in disjoint ormostly disjoint spectrum bands, as described above, can reduce, and evenpossibly eliminate, the detection of the artifact events.

In some embodiments, the computer 268 may disregard events detected attimes at which the PSOG 262 emits light to illuminate the eye. In otherembodiments, the event camera 264 may refrain from generating events attimes at which the PSOG 262 emits light to illuminate the eye.

In one embodiment, the event camera 264 is head-mounted, and a firstrange of eye positions trackable from the measurements 263 is broaderthan a second range of eye positions trackable from the events 265.Optionally, the computer 268 calculates at least some of the eyepositions 269 based on the measurements 263 when the eye positions areoutside the second range. Optionally, the computer 268 calculates atleast some of the eye positions 269 based on the events 265 when the eyepositions are within the second range. Optionally, the computer 268 doesnot utilize the measurements 263 to calculate the eye positions 269 whenthe eye positions are within the second range.

In some embodiments, the PSOG 262 and the event camera 264 may beutilized together to train and/or calibrate a model for detecting eyemovement and/or eye positions based on measurements taken by PSOG. Insome embodiments, data obtained from the event camera 264 (e.g., byanalyzing images generated from the events detected by the event camera264) is utilized as a “ground truth”, indicating eye movements and/oreye positions at the time. This data can then be utilized to train amodel (e.g., by setting parameters of the model) for power-efficient eyetracking that maps measurements of reflections obtained with PSOG 262 toeye movements and/or eye positions.

In one embodiment, in order to train such a model, the computer 268calculates eye positions and/or eye movements based on analysis ofevents 265 (referred to herein as “event-based positions”). Theevent-based positions serve as labels (the ground truth) for trainingthe model. Together with corresponding measurements of reflectionsmeasured at corresponding times with the PSOG 262, a training set ofsamples is generated. This training set is provided to a machinelearning training algorithm and is used to train and/or update themodel. In one example, the model may be a regression model. In anotherexample, the model may include parameters of a convolution neuralnetwork. Additional details about how such models may be trained areprovided in the reference Rigas 2017.

In some embodiments, calibration approaches known in the art fordetermining eye positions based on PSOG may be adapted to someembodiments of the system illustrated in FIG. 4 . For example, insteadof obtaining ground truth positions by providing the user with cuesindicating where to look, gaze estimation may be performed to obtain asimilar reference ground truth using one or more of the approaches ofthe references mentioned herein for gaze estimation from events detectedby an event camera.

In one embodiment, the event camera 264 captures the frame images 266 ofthe eye at a rate that is significantly lower than a rate at which themeasurements 263 of the reflections are measured by the PSOG 262.Optionally, the computer 268 processes the events 265, the measurements263, and the frame images 266 concurrently. Concurrent processing ofevents and frame images captured by an event camera is described forexample in the reference Angelopoulos, et al., “Event Based, Near EyeGaze Tracking Beyond 10,000 Hz” arXiv preprint arXiv:2004.03577 (2020).Adding the PSOG-based eye positions (which are not incremental updateslike the events) may improve the robustness of the model and may enablethe event camera 264 to lower the rate of capturing the frame images266. One example for the significantly lower frame rate includes anembodiment of a system in which the event camera 264 captures the frameimages 266 at a rate of 5 Hz, which are combined with measurements ofthe PSOG 262 that operates at a rate of 1,000 Hz.

In one embodiment, the computer 268 operates the event camera 264 in anormal mode when the EMV is above a threshold and eyelids covering theeye are open, and operates the event camera 264 in a low-power modewhile the eyelids are closed or refrains from processing the events 265while the eyelids are closed. Optionally, the computer 268 detectsopening of the eyelids based on the measurements 263. Additional detailsregarding how the computer 268 may detect opening and closing of theeyelids based on measurements of reflections obtained with PSOG areprovided below in the discussion regarding embodiments illustrated inFIG. 10 .

In one embodiment, the computer 268 utilizes the eye positions 269,which are calculated based on the events 265 and the measurements 263,for calibration in calculations of eye positions based on themeasurements 263. Optionally, in this embodiment, calculating eyepositions based on the measurements 263 consumes less power compared tocalculating eye positions based on the events 265. Optionally, theevents 265 are utilized to calculate ground truth eye positions usingone or more of the computational approaches mentioned herein. Theseground truth eye positions are used to generate labels for trainingsamples for training and/or calibrating a model for determining eyepositions based on measurements of reflections measured with PSOG, asdiscussed above.

The following method may be used by systems modeled according to FIG. 4. The steps described below may be performed by running a computerprogram having instructions for implementing the method. Optionally, theinstructions may be stored on a computer-readable medium, which mayoptionally be a non-transitory computer-readable medium. In response toexecution by a system including a processor and memory, the instructionscause the system to perform the following steps:

In Step 1, emitting light and taking measurements of reflections of thelight from an eye of a user by a photosensor-oculography device (PSOG).For example, the PSOG 262 may be utilized in this step to obtain themeasurements 263 of the reflections of the eye of the user.

In Step 2, capturing the eye and providing events by an event camera.For example, the event camera 264 may be utilized in this step to detectthe events 265.

And in Step 3, calculating positions of the eye (referred to herein as“eye positions”) based on the events and measurements of thereflections. For example, the computer 268 may be utilized in this stepto calculate the eye positions 269 based on the measurements 263 and theevents 265.

Calculating the eye positions in Step 3 based on the measurements of thereflections and the detected events may be done in different ways. Inone embodiment, the method may optionally include a step involvingutilizing the events to incrementally update eye positions calculatedbased on the measurements of the reflections. In another embodiment, themethod may optionally include the following steps: calculating based onthe measurements of the reflections a stream of main eye positions, andcalculating based on the events intermediate streams of eye positions,which are located between at least some of the main eye positions.Optionally, on average, time that elapses between calculation ofconsecutive eye positions in the intermediate streams is much shorterthan average time that elapses between calculation of consecutive eyepositions from among the main eye positions.

In one embodiment, a first range of eye positions trackable frommeasurements of the reflections measured by the PSOG in step 1 isbroader than a second range of eye positions trackable from the eventsdetected by the event camera in Step 2, and the method includes a stepof calculating the eye positions based on the measurements of thereflections when the eye positions are outside the second range.

In one embodiment, the method may optionally include the followingsteps: calculating eye positions and/or eye movements based on analysisof the events detected in Step 2 (referred to herein as event-basedpositions), generating labels based on the event-based positions,generating feature values based on measurements of the reflections(measured by the PSOG in Step 1), and providing the labels and featurevalues to train a machine learning-based model for detecting eyepositions and/or eye movements based on measurements of reflections.

In one embodiment, the method may optionally include a step ofcapturing, by the event camera, frame images of the eye at a rate thatis significantly lower than a rate at which the reflections are measuredby the PSOG. Optionally, processing the events, measurements of thereflections, and the frame images is done concurrently.

In one embodiment, the method may optionally include the followingsteps: operating the event camera in a normal mode during saccades,operating the event camera in a low-power mode during fixations orrefraining from processing the events during the fixations, andcalculating the eye positions during the fixations based on themeasurements of the reflections.

In one embodiment, the method may optionally involve operating the eventcamera in a normal mode during saccades, operating the event camera in alow-power mode while the eyelids are closed or refraining fromprocessing the events while the eyelids are closed, and detectingopening of the eyelids based on measurements of the reflections.

FIG. 5 illustrates an embodiment of an eye tracking system. In someembodiments, the eye tracking system includes a photosensor-oculographydevice (PSOG 272), a head-mounted camera 274, and a computer 278. ThePSOG 272 is configured to emit light and to take measurements 273 ofreflections of the light from an eye of a user. The head-mounted camera274 is configured to capture images of the eye utilizing a sensor (i.e.,an image sensor) that supports changing of its region of interest (ROI).Optionally, one or more components of some embodiments of the systemillustrated in FIG. 5 , such as the PSOG 272, the head-mounted camera274, head-mounted light sources 271, and/or the computer 278 may becoupled to, and/or embedded in, frames of smartglasses 270 that areconfigured to be worn on the user's head.

The computer 278 calculates a position of the eye (also referred toherein as an “eye position”) based on the measurements 273 of thereflections. The computer 278 then utilizes the calculated eye positionto place an ROI 275 around pixels covering the eye's pupil, and readsthe ROI 275 from the head-mounted camera 274. Optionally, reading theROI 275 involves generation of images 276 that include the pixelscovered by the ROI 275. Optionally, the images 276 do not include pixelsthat are not covered by the ROI 275, so for example, these pixels may beconsidered “captured” but not “read”. Optionally, the ROI 275 coversless than 25% of the field of view of the sensor used by thehead-mounted camera 274 (which is the sensor that supports the changingof its region of interest). Optionally, the ROI 275 covers less than 50%of the field of view of the sensor used by the head-mounted camera 274.In some embodiments, to calculate the eye position, the computer 278 mayutilize one or more of the techniques mentioned herein for calculatingeye positions and/or eye movements based on the measurements ofreflections obtained by PSOG, such as the techniques mentioned in Rigas2017, Rigas 2018, and/or standalone PSOG approaches known in the art,such as the approaches mentioned further above in the references (i)Zemblys et al. (2018), (ii) Katrychuk et al. (2019), and/or (iii) Li etal. (2020).

In CMOS-based camera sensors, such as the image sensors that may be usedby the head-mounted camera 274 in some embodiments, the term “region ofinterest” (ROI) may also be known as: window of interest readout,windowing, sub-windowing, region of interest readout, programmableregion of interest, area of interest, partial readout window, randompixel access, and direct pixel addressing. In CCD-based camera sensors,the term region of interest may also be known as partial scanning For “asensor that supports changing of its ROI”, the changing of the ROI is afeature that allows reading only a portion of the pixels that werecaptured, and by that increasing the readout speed of the ROI, andoptionally also reducing the camera's duty cycle. Some sensors alsoallow multiple ROI readouts in order to simplify the operation ofmultiple windowing. Sentences of the form of “set the ROI according to asubset of pixels”, “to place the ROI around pixels covering an object”,or “to place the ROI around pixels covering pupil of the eye” refer tosetting the coordinates of the ROI to cover the “subset of pixels”,“pixels covering an object”, or “pupil of the eye”, respectively.Herein, pixels are considered to “cover” a region/object if they areable to detect light reflected from that region/object.

The eye tracking system illustrated in FIG. 5 may be viewed as avideo-oculography (VOG) system that utilizes photosensor-oculography(PSOG) in order to operate the camera used for the VOG more efficientlyand thus be able to save power involved in eye tracking and/or increasethe frequency at which images are acquired (due to the smaller number ofpixels that needs to be read). Embodiments of the system illustrated inFIG. 5 may be utilized to implement different types of eye trackers, asdiscussed in more detail below. For example, the images 276 may beprovided to an image-based eye-tracker that utilizes the images 276 tocalculate various pupil features and/or a direction of gaze of the user.

In some embodiments, utilization of the eye position to place the ROI275 around pixels covering the eye's pupil involves selecting, based onthe eye position, a subset of pixels covering less than half field ofview of the image sensor of the head-mounted camera 274, and setting theROI 275 according to the subset. Various approaches may be utilized toselect the pixels that will be covered by the ROI 275.

In one example, the eye position may be utilized to determine a certainregion of pixels such as pixels that are a certain distance from aco-ordinate of the eye position). Optionally, the ROI 275 may be set toinclude a list of pixels that fall within the certain region. In anotherexample, the ROI 275 is set according to co-ordinates of a bounding boxthat includes the eye position. Optionally, the eye position correspondsto the center of the bounding box or at a pre-determined location in thebounding box. Optionally, the bounding box has a predetermined sizeand/or location with respect to the eye position.

FIG. 6 illustrates a scenario in which the system illustrated in FIG. 5is used to read different ROIs when the eye is at different eyepositions. The figure illustrates a portion 270′ of the smartglassesframe 270 that includes the lens in front of the right eye. In thisillustration, the PSOG 272 includes several emitters and discretephotosensors disposed in the portion 270′ of the frame (e.g.,photosensor 277 a and emitter 277 b). The head-mounted camera 274 takesimages 279 that cover the area of the whole right eye. Note that thetrapezoid shape of the images 279 is intended to illustrate possibledistortions in the images due to the acute angle between the opticalaxis of the head-mounted camera 274 and the area of the eye. The eyeposition as calculated based on measurements of the reflections of theeye is used to change the ROIs read from the head-mounted camera 274.When the pupil is oriented outwards (i.e., when the user is looking tothe right), as illustrated on the left side of FIG. 6 , a first eyeposition is calculated and a first ROI (ROL) is set accordingly, and animage 276 a of ROIs is read from the head-mounted camera 274. And whenthe eye moves, and the pupil is oriented more inwards, as illustrated onthe right side of FIG. 6 , a second eye position is calculated and asecond ROI (ROI₂) is set accordingly, and an image 276 b of ROI₂ is readfrom the head-mounted camera 274

Knowing the eye position can be utilized in some embodiments to identifylocations of certain eye features referred to herein as “advantageousfeatures” prior to capturing the images. These advantageous features maybe useful for calculating various pupil features discussed herein. Someexamples of advantageous features include pupil location, iris location,limbus location, and identification of locations of one or more glintsthat are expected to appear on the cornea from the camera's point ofview. In some embodiments, the computer 278 is utilizes the eye positionto select advantageous features before capturing the images 276. Forexample, locations of the advantageous features relative to the eyeposition are determined from evaluation of images (that are notrestricted to the ROI 275) captured by the head-mounted camera 274. Thecomputer 278 then sets the ROI 275 to cover the locations ofadvantageous features, and provides properties of said advantageousfeatures to an image-based eye tracker (which may optionally involve thecomputer 278 performing calculations on the images 276).

Different positions of the eye expose to the camera different parts ofthe eye from different angles, which causes different features to havedifferent quality levels for the VOG. The eyelid position and thelighting conditions also affect the efficiency and accuracy of VOG.Therefore, the computer 278 can utilize the PSOG's data (such as eyeposition, eyelid position, illumination) to select the advantageousfeatures before capturing image, then set the ROI to capture theadvantageous features, and then provide the properties of saidadvantageous features to the VOG tracker. The metric for selecting theadvantageous features may depend on the expected accuracy to be obtainedfrom analyzing the features (the higher the accuracy the better thefeature is), and/or the expected required processing power to captureand/or analyze the feature (the lower the processing power the betterthe feature is).

In one embodiment, the image-based eye tracker comprises a limbus-basedeye tracker and a glint-based eye tracker; switching between utilizationof these two trackers is performed automatically based on theadvantageous features that were selected, and a metric for selecting theadvantageous features comprises at least one of expected accuracy to beobtained from analyzing the advantageous features and expected requiredprocessing power to capture and/or analyze the advantageous features.

Some implementations of eye trackers are based on detecting locations ofglints in images that are caused when light from known light sources isreflected from the eye. For example, with VOG calculation the gazedirection can rely on the pupil-glint vectors, which are the relativedistances between the centers of the pupil and one or more cornealreflections. The number of glints utilized, may depend on the number oflight sources, and one example of an eye tracking process can be dividedinto two stages (i) analyzing the images to locate features, such as theglints and the pupil centers, and (ii) estimating the gaze based on thedetected features.

There are various eye tracking approaches known in the art that rely ondetecting glints (these systems are referred to herein as theglint-based eye trackers” mentioned above). One example for glint-basedeye tracking is described in Mestre, et al., “Robust eye tracking basedon multiple corneal reflections for clinical applications.” Journal ofbiomedical optics 23.3 (2018): 035001. Additional examples ofglint-based eye tracking are provided in Hosp, Benedikt, et al.“RemoteEye: An open-source high-speed remote eye tracker: Implementationinsights of a pupil-and glint-detection algorithm for high-speed remoteeye tracking.” Behavior research methods 52.3 (2020).

In some embodiments, the system illustrated in FIG. 5 may be utilized toperform glint-based eye tracking. In these embodiments, head-mountedlight sources 271 are configured to emit light that generates glints onthe eye. The computer 278 utilizes the eye position (which is calculatedbased on the measurements 273 of the reflections) to select a subset ofthe head-mounted light sources 271 that are expected to generate one ormore glints on the cornea. The computer 278 then operates the subset ofthe head-mounted light sources 271 at a higher intensity compared to therest of the light sources. For example, the subset of the head-mountedlight sources 271 can be selected based on the eye position calculatedfrom the measurements 273, in such a way that when the eye positionchanges, the subset of the head-mounted light sources 271 may changeaccordingly (so they produce desired glints on the eye). It is notedthat operating the subset of the head-mounted light sources 271 at ahigher intensity compared to the rest of the head-mounted light sources271 also refers to not operating the rest of the head-mounted lightsources 271.

In some embodiments, the arrangement of the head-mounted light sources271 and their operation (e.g., not all the light sources must beoperated all the time) may have an effect on the glints that aregenerated. For example, different light sources may be interfered by theupper and/or lower eyelids as a function of the eyelid positions. Themost suitable light sources to emit the light depend on both the eyetracker setup (which includes considerations such as the location ofcamera, the locations of the light sources, and the locations of thephotosensors) and the properties of the user (which includesconsiderations such as the anatomical shape of eyelids, ethnicity, andage). Therefore, taking in account said considerations can have asignificant impact on the performance of the eye tracker.

In one embodiment, the computer 278 calculates positions of the eyelidsbased on the measurements 273 of the reflections, and utilizes thepositions of the eyelids to select a subset of the head-mounted lightsources 271 that are expected to generate one or more glints on an areaof the cornea not covered by the eyelids. The computer 278 then operatesthe subset of the head-mounted light sources 271 at a higher intensitycompared to the rest of the light sources. Optionally, when the eyeposition changes, the calculations above are reperformed, and the subsetof the head-mounted light sources 271 may change accordingly.

In another embodiment, the computer 278 calculates positions of theeyelids based on the measurements 273 of the reflections, and utilizesthe positions of the eyelids to select a subset of the head-mountedlight sources 271 that are not expected to be interfered by the eyelidsand/or eyelashes, and/or expected to have a minimal interference withthe eyelids and/or eyelashes. Such interference can cause light emittedfrom light sources not to form expected glints due to the at least someof the emitted light being absorbed by the eyelids and/or eyelashes. Thecomputer 278 may then operate the subset of the head-mounted lightsources 271 at a higher intensity compared to an intensity at which therest of the light sources, from among the head-mounted light sources271, are operated.

By knowing the location of the head-mounted camera 274, the locations ofthe head-mounted light sources 271, and the eye position, in someembodiments, the computer 278 estimates the locations of the glints onthe cornea, and sets the ROI 275 to be around the cornea. Optionally,the properties of the ROI 275 are fed into the glint-based eye trackerthat processes the ROI 275 in a similar manner to processing croppedimages together with their cropping parameters, as is performed by someeye trackers known in the art. To this end, in one embodiment, thecomputer 278 calculates positions of the eyelids based on themeasurements 273 of the reflections, utilizing the eye position and thepositions of the eyelids to set the ROI 275 around the cornea, such thatthe ROI 275 covers an area that is not greater than two times the areaof the eye that is not covered by the eyelids. The images 276 then canbe provided for utilization of the glint-based eye tracker. In anotherembodiment, the computer 278 utilizes the eye position to set the ROI275 around the cornea, such that the ROI 275 covers an area that is notgreater than two times the area of a square that surrounds the iristightly.

In one embodiment, the computer 278 operates an eye tracker that isbased on detecting pupil contour. In this embodiment, the computer 278utilizes the eye position to set the ROI 275 around the iris, such thatthe ROI 275 covers an area that is not greater than two times the areaof a square that surrounds the iris tightly. Optionally, determining howto set the ROI 275 to include the iris is done using a model trainedbased on images captured by the head-mounted camera 274 (in which thepupil contour is detected and an appropriate square is positioned), asdiscussed above.

In another embodiment, the computer 278 operates an eye tracker that isbased on detecting pupil contour. In this embodiment, the computer 278estimates a location of the pupil based on the eye position, andutilizes the eye position to set the ROI 275 around the pupil, such thatthe ROI 275 covers an area that is not greater than two times the areaof the eye that is not covered by the eyelids.

Some of the embodiments described herein involve calculating variouseye-related features based on the eye position and/or the measurements273 of the reflections, and then utilizing these features to determineoperational parameters, such as selection of a subset of thehead-mounted light sources 271 and/or selection of the ROI 275 so itconforms to a certain condition (like including certain features). Someexamples of eye-related features that may be calculated for thesepurposes include detection of locations of the eyelids and/or eyelashes, calculation of a pupil location, an iris location, limbuslocation, and estimation of locations of one or more glints that areexpected to appear on the cornea from the camera's point of view. Thesefeatures are often more easily and/or accurately extracted from images(e.g., images taken by the head-mounted camera 274). However, asexplained below, by using such images to train a machine learning-basedmodel, it is possible to calculate values of these eye-related features,even at times at which images are not captured by the head-mountedcamera 274, which can help conserve power (since the head-mounted camera274 can be operated for shorter periods).

In some embodiments, eye positions and/or the measurements of thereflections obtained by the PSOG 272 (from which the eye positions arecalculated) are an input to a function that calculates values of aneye-related feature, such as one or more of the eye-related featuresmentioned above. The calculation of the values of the eye-relatedfeature may utilize, in these embodiments, a model that is trained withsamples that each includes feature values that are generated based on aneye position and/or measurements used to calculate the eye position anda label generated from analysis of one or more images captured by thehead-mounted camera 274 at the same time the measurements used togenerate the feature values were taken. Examples of the feature valuesthat may be generated by the computer 278 include co-ordinates of theeye position, dynamics of the eye position (e.g., changes to speed andtrajectory), and various PSOG-related feature values that are known inthe art mentioned in references described herein. Additionally, thecomputer 278 may generate feature values indicating which of thehead-mounted light sources 271 were utilized in order to generate themeasured reflections.

The labels are generated using image analysis techniques known in theart to detect the features that are to be calculated with the model,such as analyzing images to find the location of glints or positions ofeyelids (which are more readily detected the images compared to the PSOGdata).

Training samples that include the aforementioned features and labels areprovided to a machine learning algorithm, such as an algorithm fortraining a regression model, a neural network, etc. This results in apersonalized model that is capable of calculating eye-related featuresof the user based on measurements of the user obtained with the PSOG272.

In one example, the computer 278 (i) extracts a set of eyelid positionsbased on analyzing the images 276, and (ii) provides labels that arebased on the set of eyelid positions, and corresponding feature valuesthat are based on the measurements 273 of the reflections, to train amodel for calculating eyelid position based on additional measurementsof the reflections obtained by the PSOG 272 (corresponding to times inwhich images are not captured by the head-mounted camera 274).

Since calculation of pupil feature may be computationally intensive, insome embodiments, the computer 278 my determine whether such acalculation is necessary prior to performing it. To this end, thecomputer 278 may calculate a correlation between a current image takenby the head-mounted camera 274 and a previous image having the same ROI275 and eye position. If the correlation is above a certain threshold,the computer 278 retrieves a previously calculated value of a pupilfeature when the correlation exceeds a threshold (i.e., use a “cached”value), or calculate a new value for the pupil feature based on thecurrent image when the correlation is below the threshold.

The following method may be used by systems modeled according to FIG. 5. The steps described below may be performed by running a computerprogram having instructions for implementing the method. Optionally, theinstructions may be stored on a computer-readable medium, which mayoptionally be a non-transitory computer-readable medium. In response toexecution by a system including a processor and memory, the instructionscause the system to perform the following steps:

In Step 1, emitting light and taking measurements of reflections of thelight from an eye of a user by a photosensor-oculography device (PSOG).For example, the PSOG 272 may be utilized in this step to obtain themeasurements 273 of the reflections of the eye of the user.

In Step 2, capturing images of the eye by a head-mounted camera thatutilizes a sensor that supports changing of its region of interest(ROI). For example, the head-mounted camera 274 may be utilized tocapture the images in this step.

In Step 3, calculating, based on measurements of the reflectionscaptured in Step 2, a position of the eye (referred to herein as an “eyeposition)”.

In Step 4, utilizing the eye position for placing the ROI around pixelscovering the eye's pupil.

And in Step 5, reading the ROI from the camera. Optionally, data read inthis step is used calculate one or more pupil features.

In one embodiment, the method may optionally include the followingsteps: utilizing the eye position to select advantageous features beforecapturing the images, setting the ROI to cover the advantageousfeatures, and providing properties of said advantageous features to animage-based eye tracker. Optionally, the advantageous features comprisedata indicative of at least one of the following: pupil location, irislocation, limbus location, and which glints are expected to appear onthe cornea from the camera's point of view.

In another embodiment, the method may optionally include the followingsteps: (i) utilizing the eye position for selecting a subset ofhead-mounted light sources that are expected to generate one or moreglints on the cornea (e.g., the head-mounted light sources 271), and(ii) operating the subset of the head-mounted light sources at a higherintensity compared to the rest of the light sources that are configuredto emit light toward the eye.

In yet another embodiment, the method may optionally include thefollowing steps: (i) calculating positions of the eyelids based on themeasurements of the reflections, (ii) utilizing the positions of theeyelids for selecting a subset of the light sources that are expected togenerate one or more glints on an area of the cornea not covered by theeyelids, and (ii) operating the subset of the light sources at a higherintensity compared to the rest of the light sources that are configuredto emit light toward the eye.

In still another embodiment, the method may optionally include thefollowing steps: calculating positions of the eyelids based on themeasurements of the reflections, (ii) operating a glint-based eyetracker, and (iii) utilizing the eye position and the positions of theeyelids for setting the ROI around the cornea, such that the ROI coversan area that is not greater than two times the area of the eye that isnot covered by the eyelids

The approach of the eye tracking system illustrated in FIG. 5 may beexpanded to involve additional types of sensors and/or devices (insteadof PSOG or in addition to it) that provide information about the eyeposition. This information from the additional types of sensors and/ordevices can then be utilized to operate a head-mounted camera that ispart of the eye tracking system more efficiently by setting the ROI to arelevant region of the field of view of the head-mounted camera.

FIG. 7 illustrates an embodiment of an eye tracking system. In oneembodiment, the eye tracking system includes a head-mounted device 282,the head-mounted camera 274, and a computer 288. The head-mounted device282 takes measurements 283 that are indicative of a position of an eyeof a user (referred to herein as “eye position”). The head-mountedcamera 274 captures images of the eye utilizing a sensor (i.e., an imagesensor) that supports changing of its region of interest (ROI).Optionally, one or more components of some embodiments of the systemillustrated in FIG. 7 , such as the head-mounted device 282, thehead-mounted camera 274, the head-mounted light sources 271, and/or thecomputer 288 may be coupled to and/or embedded in frames of smartglasses280, which are configured to be worn on the user's head.

The computer 288 calculates the eye position based on measurements 283.The computer 288 then utilizes the calculated eye position to place anROI 285 around pixels covering the eye's pupil, and reads the ROI 285from the head-mounted camera 274 (e.g., in order to obtain image 286 ofthe ROI 285). Optionally, the ROI 285 covers less than 30% of the fieldof view of the sensor used by the head-mounted camera 274 (which is thesensor that supports the changing of its region of interest).Optionally, the ROI 285 covers less than 50% of the field of view of thesensor used by the head-mounted camera 274.

To calculate the eye position based on the measurements 283, thecomputer 288 may utilize various approaches known in the art fordetermining the eye position which are appropriate for the type ofsensors used in the head-mounted device 282 (as discussed below).Additionally or alternatively the computer 288 may utilize a machinelearning-based approach, as discussed further below.

The head-mounted device 282 may include different types of components inembodiments described herein. These different components may providedifferent types of measurements indicative of the eye positions.

In one embodiment, the head-mounted device 282 includes anelectrooculography (EOG) device, and measurements 283 include a value ofan electrical potential between electrodes placed close to the eye.Calculating the eye position in this embodiment may involve varioustechniques known in the art for calculating eye position based onelectrical potentials. Some examples of techniques that may be used bythe computer 288 to calculate the eye position based on EOG data aredescribed in Barea, Rafael, et al. “Wheelchair guidance strategies usingEOG.” Journal of intelligent and robotic systems 34.3 (2002): 279-299.In one example, a personalized machine learning-based model, which istrained based on images captured by the head-mounted camera 274 alongwith EOG measurements of the user taken at the same time, may beutilized to calculate the eye position, as discussed below. Embodimentsin which the head-mounted device 282 include an EOG device can be usedto implement a hybrid EOG-camera eye tracker, in which the eye positionis calculated based on the electrical potential between the electrodesplaced close to the eye.

In another embodiment, the head-mounted device 282 includes anelectromyography (EMG) device, and the measurements 283 include a valueof an electrical potential generated by muscle cells. Calculating theeye position in this embodiment may involve various techniques known inthe art for calculating eye position based on electrical potentials.Some examples of techniques that may be used by the computer 288 incalculation of the eye position based on EMG data are described inAhsan, et al., “EMG signal classification for human computerinteraction: a review.” European Journal of Scientific Research 33.3(2009): 480-501, and in Monaco, A., et al. “Ocular correction effects onEMG activity of stomatognathic muscles in children with functionalmandibular lateral-deviation: a case control study.” Eur J Paediatr Dent7.2 (2006): 81-8. In one example, a personalized machine learning-basedmodel, which is trained based on images captured by the head-mountedcamera 274 along with EMG measurements of the user taken at the sametime, may be utilized to calculate the eye position, as discussed below.Embodiments in which the head-mounted device 282 includes an EMG devicecan be used to implement a hybrid EMG-camera eye tracker, in which theeye position is calculated based on the electrical potential generatedby the muscle cells.

In yet another embodiment, the head-mounted device 282 includes anoptical flow sensor, the measurements 283 include values of optical flowand/or visual motion, and the eye position is calculated using anoptical flow algorithm known in the art. Some examples of techniquesthat may be used by the computer 288 in calculation of the eye positionbased on optical flow data are described in Jiménez-Pinto, et al.,“Optical flow and driver's kinematics analysis for state of alertsensing.” Sensors 13.4 (2013): 4225-4257. In one example, the opticalflow sensor is an image sensor configured to measure optical flow and/orvisual motion, and the motion of the eye is calculated based on anoptical flow algorithm.

In still another embodiment, the head-mounted device 282 includes arange sensor, and the measurements 283 include values of a range (e.g.,distance) between the range sensor and the eye. A range sensor may be asensor configured to detect the presence of nearby objects withoutphysical contact, such as a Doppler sensor, a passive optical sensor, aninfrared sensor, a radar, and a sensor that measures time of flight.Since the eyeball is not perfectly spherical, the proximity between therange sensor at a fixed distance from the user's skull and the portionof the eyeball in the sensor's direct line-of-sight changes with eyemovement. For example, the cornea is raised relative to the sclera, thusa shorter detected range may indicate that the cornea is in the sensor'sdirect line-of-sight. In one example, the eye position may be calculatedutilizing a personalized machine learning-based model, which is trainedbased on images captured by the head-mounted camera 274 along with rangemeasurements taken with the range sensor at the same time. Embodimentsin which the head-mounted device 282 includes a range sensor can be usedto implement a hybrid range-camera eye tracker that calculates the eyeposition based on the range between the range sensor and the eye.

In some embodiments, the head-mounted device 282 includes a PSOGconfigured to emit light and take measurements of the reflections of thelight from the eye (e.g., the PSOG 272), and the eye position iscalculated based on the measurements of the reflections.

Various embodiments described herein involve calculating the eyeposition based on the measurements 283, which may include differenttypes of data, depending on the types of sensors and/or devices includedin the head-mounted device 282 (e.g., EOG, EMG, PSOG, etc.). Determiningthe eye position can often be done accurately from analysis of images,such as images captured by the head-mounted camera 274. Additionally,determining the eye position from images at the same frequency asdetermined from the measurements taken by the head-mounted device 282will typically be more power and computationally intensive than usingthe head-mounted device 282 alone. The fact that the head-mounted camera274 and the head-mounted device 282 can be operated at the same time canbe utilized to collect training data that can be used to calibratemodels used to calculate eye position based on measurements taken by thehead-mounted device 282, and thus achieve calculation of the eyeposition that is both accurate and power-efficient.

In some embodiments, measurements obtained from the head-mounted device282 are an input to a function that calculates the eye positionutilizing an eye position model. The eye position model is trained withsamples that each includes feature values that are generated based oncertain measurements obtained from the head-mounted device 282 at acertain time (e.g., features known in the art that are used to determineeye position from the specific type of data in the measurements), and alabel indicative of the eye position at the certain time, as determinedfrom analysis an image captured by the head-mounted camera 274 at thecertain time. The labels are generated using image analysis techniquesknown in the art to detect the eye position.

Training samples that include the aforementioned features and labels areprovided to a machine learning algorithm, such as an algorithm fortraining a regression model, a neural network, etc. This results in theeye position model being a personalized model that is capable ofcalculating eye position of the user based on measurements of the userobtained with the head-mounted device 282.

As discussed above, with respect to the computer 278, knowing the eyeposition can be utilized in some embodiments to identify locations ofcertain eye features, referred to herein as “advantageous features”,prior to capturing an image with the head-mounted camera 274. In oneembodiment, the computer 288 utilizes the eye position, which iscalculated based on the measurements 283 that are indicative of the eyeposition, to select advantageous features before capturing the image286. The computer 288 sets the ROI 285 to cover the advantageousfeatures, and provides properties of said advantageous features to animage-based eye tracker. Optionally, the advantageous features includedata indicative of at least one of the following: pupil location, irislocation, limbus location, and which glints are expected to appear onthe cornea from the camera's point of view.

In some embodiments, the system illustrated in FIG. 7 may be utilized toperform glint-based eye tracking. In these embodiments, the head-mountedlight sources 271 are configured to emit light that generates glints onthe eye. The computer 288 utilizes the eye position, which is calculatedbased on the measurements 283, to select a subset of the head-mountedlight sources 271 that are expected to generate one or more glints onthe cornea. The computer 288 then operates the subset of thehead-mounted light sources 271 at a higher intensity compared to therest of the light sources. For example, the subset of the head-mountedlight sources 271 can be selected based on the eye position, in such away that when the eye position changes, the subset of the head-mountedlight sources 271 may change accordingly (so they produce desired glintson the eye). It is noted that operating the subset of the head-mountedlight sources 271 at a higher intensity compared to the rest of thehead-mounted light sources 271 (at a certain time) also refers to notoperating the rest of the head-mounted light sources 271 (at the certaintime).

In one embodiment, the computer 288 operates an eye tracker that isbased on detecting pupil contour. In this embodiment, the computer 288utilizes the eye position to set the ROI 285 around the iris, such thatthe ROI 285 covers an area that is not greater than two times the areaof a square that surrounds the iris tightly. Optionally, determining howto set the ROI 2875 to include the iris is done using a model trainedbased on images captured by the head-mounted camera 274 (in which thepupil contour is detected and an appropriate square is positioned aroundiris), as discussed herein.

The following method may be used by systems modeled according to FIG. 7. The steps described below may be performed by running a computerprogram having instructions for implementing the method. Optionally, theinstructions may be stored on a computer-readable medium, which mayoptionally be a non-transitory computer-readable medium. In response toexecution by a system including a processor and memory, the instructionscause the system to perform the following steps:

In Step 1, taking, with a head-mounted device (e.g., the head-mounteddevice 282), measurements indicative of a position of an eye of a user(eye position).

In Step 2, capturing an image of the eye by a head-mounted camera thatutilizes a sensor that supports changing of its region of interest(ROI). For example, the head-mounted camera 274 may be utilized tocapture the image in this step.

In Step 3, calculating the eye position based on the measurements takenin Step 1.

In Step 4, utilizing the eye position for placing the ROI around pixelscovering pupil of the eye.

And In Step 5, reading the ROI from the head-mounted camera.

Another way in which the eye position may be used to improve efficiencyof eye tracking, especially with video-based eye trackers, is used byembodiments of eye tracking systems illustrated in FIG. 8 . In someembodiments, an eye tracking system includes the head-mounted device282, a head-mounted camera 294, and a computer 298. The head-mounteddevice 282 takes the measurements 283 that are indicative of a positionof an eye of a user (referred to herein as “eye position”). Thehead-mounted camera 294 captures an image 296 of the eye of the user.Optionally, one or more components of some embodiments of the systemillustrated in FIG. 8 , such as the head-mounted device 282, thehead-mounted camera 294, the head-mounted light sources 271, and/or thecomputer 298 may be coupled to and/or embedded in frames of smartglasses290 that are configured to be worn on the user's head.

The computer 298 calculates the eye position based on the measurements283 that are indicative of the eye position. The computer 298 thenutilizes the calculated eye position to crop the image 296 around thepupil to produce cropped image 297, which is provided to a video-basedeye tracker 299 (e.g., for more advanced eye tracking analysis).Optionally, the size of the cropped image 297 is less than a third ofthe size of its respective uncropped image (the image 296). Optionally,the cropped image 297 covers an area that is not greater than two timesthe area of a square that surrounds the iris tightly. Optionally,calculating the eye position based on the measurements 283 is performedat a rate that is at least ten times higher than the rate of capturingthe image by the head-mounted camera 294. Optionally, when cropping theimage 296, the computer 298 utilizes a similar approach to the oneutilized for setting the ROI 275 or the ROI 285, discussed above.Namely, the computer 298 crops the image so it includes a sufficientmargin, relative of the eye position, such that the cropped image 297includes the pupil of the eye.

In one embodiment, the video-based eye tracker 299 calculate, based onthe cropped image 297, at least one of the following values: a pupildiameter, and pupillary response. The term “pupillary response” refersto at least one of dilation response (widening of the pupil, also knownas pupil dilation) and constriction response (narrowing of the pupil).To perform the calculation of the at least one of the pupil diameter andthe pupillary response, the video-based eye tracker 299 may utilize oneor more of the techniques known in the art for calculating pupildiameter and/or pupillary response from images of the eye.

As stated above in the discussion regarding embodiments of FIG. 7 , thehead-mounted device 282 may include different types of components, whichis also the case in some embodiments of the system illustrated in FIG. 8, as described below. These different components may provide differenttypes of values that are indicative of the eye positions that can beused by the computer 298 in order to produce the cropped image 297.

In one embodiment, the head-mounted device 282 includes a PSOG, themeasurements 283 include measurements of reflections of light emitted bythe PSOG towards the eye. Optionally the system further includes anotherPSOG that emits light and measures reflections of the light from theother eye of the user, and the system may also include anotherhead-mounted camera that captures an image of at least a portion of theother eye of the user. Optionally, the head-mounted cameras may be partof a video oculography system.

In another embodiment, the head-mounted device 282 includes anelectrooculography (EOG) device, the measurements 283 include a value ofan electrical potential between electrodes placed close to the eye. Inyet another embodiment, the head-mounted device 282 includes anelectromyography (EMG) device, and the measurements 283 include a valueof an electrical potential generated by muscle cells. In still anotherembodiment, the head-mounted device 282 includes an optical flow sensor,the measurements 283 include values of optical flow and/or visualmotion, and the eye position is calculated using an optical flowalgorithm known in the art. And in yet another embodiment, thehead-mounted device 282 includes a range sensor, and the measurements283 include a value of a range (e.g., distance) between the range sensorand the eye.

In some embodiments, the system illustrated in FIG. 8 may be utilized toperform glint-based eye tracking. In these embodiments, the head-mountedlight sources 271 emit light that generates glints on the eye. Similarlyto the computer 288, the computer 298 may utilize the eye position,which is calculated based on the measurements 283, to select a subset ofthe head-mounted light sources 271 that are expected to generate one ormore glints on the cornea. The computer 298 then operates the subset ofthe head-mounted light sources 271 at a higher intensity compared to therest of the light sources. Additionally or alternatively, the computer298 may calculate positions of the eyelids based on the measurements283. For example, the head-mounted device 282 may include a PSOG and thecomputer 298 may utilize the approach described with respect to thecomputer 278, which may calculate the positions of the eyelids based onthe measurements 273 of the reflections. The computer 298 may thenutilize the positions of the eyelids to select a subset of thehead-mounted light sources 271 that are expected to generate one or moreglints on an area of the cornea not covered by the eyelids, and operatethe subset of the head-mounted light sources 271 at a higher intensitycompared to the rest of the head-mounted light sources 271.

The following method may be used by systems modeled according to FIG. 8. The steps described below may be performed by running a computerprogram having instructions for implementing the method. Optionally, theinstructions may be stored on a computer-readable medium, which mayoptionally be a non-transitory computer-readable medium. In response toexecution by a system including a processor and memory, the instructionscause the system to perform the following steps:

In Step 1, taking, with a head-mounted device (e.g., the head-mounteddevice 282), measurements indicative of a position of an eye of a user(eye position).

In Step 2, capturing an image of the eye by a head-mounted camera (e.g.,the head-mounted camera 294).

In Step 3, calculating the eye position based on the measurements takenin Step 1. Optionally, calculations of the eye position in this step areperformed at a rate that is at least ten times higher than the rate atwhich images are captured by the head-mounted camera in Step 2.

In Step 4, utilizing the eye position to crop the image captured in Step2 around the pupil.

And in Step 5, providing the cropped image to a video-based eye tracker.

In one example, the head-mounted device used in Step 1 includes aphotosensor-oculography device (PSOG), and the measurements taken inStep 1 are of reflections of light emitted by the PSOG towards the eye.

In another example, the head-mounted device used in Step 1 includes anelectrooculography device, and the measurements taken in Step 1 includea value of an electrical potential between electrodes placed close tothe eye.

In yet another example, the head-mounted device used in Step 1 includesan electromyography device, and the measurements taken in Step 1 includea value of an electrical potential generated by muscle cells.

FIG. 9A illustrates an embodiment of an eye tracking system. In oneembodiment, the eye tracking system includes a head-mounted device 302,a head-mounted camera 304, and a computer 308. The head-mounted device302 takes measurements 303, which are indicative of positions of an eyeof a user (referred to herein as “eye positions”). The head-mountedcamera 304 captures images 305 of the eye. Optionally, a first range ofeye positions trackable from the images 305 is narrower than a secondrange of eye positions trackable from the measurements 303. Optionally,one or more components of some embodiments of the system illustrated inFIG. 9A, such as the head-mounted device 302, the head-mounted camera304, a second head-mounted camera 306, and/or the computer 308 may becoupled to and/or embedded in frames of smartglasses 300 that areconfigured to be worn on the user's head.

Herein, an eye position is considered “trackable”, e.g., based on atleast some of the images 305 and/or at least some of the measurements303, if the respective source of data (aforementioned images and/ormeasurements) is sufficient in order for the computer 308 to determinethe eye position. Optionally, an eye position is trackable from thesource of data (aforementioned images and/or measurements) if the eyeposition is determined with a predetermined level of accuracy. Thus,certain eye positions that are not determined with the predeterminedlevel of accuracy (e.g., because the eye may be gazing farther to one ofthe sided) are not considered trackable. In one example, an eye positionis considered “trackable” if with probability of at least 95% the eyeposition is accurate within 1°.

Having the first range of eye positions be narrower than the secondrange may mean different things in different embodiments. In oneexample, the first range is contained in the second range; thus, everyeye position that falls in the first range also falls in the secondrange, but there are at least some eye positions that fall in the secondrange do not fall in the first range (i.e., they are trackable with themeasurements 303, but not in the images 305). In another example, eachof the first and second ranges corresponds to a width of the field ofview of the eye and/or each of the first and second ranges correspondsto a certain angular portion of the field of view. Optionally, the eyepositions that fall outside the first range span at least 20° of theeye's field of view. In yet another example, each eye position maycorrespond to a vector describing the direction at which the pupil ispointed. In this example, having the first range be narrower that thesecond range may mean that when these directions are mapped to a surfaceof a unit sphere, the area on the face of the unit sphere thatcorresponds to eye positions in the first range is smaller than the areaon the face of the unit sphere that corresponds to eye positions in thesecond range.

In one embodiment, the eye being tracked with the head-mounted device302 is the right eye of the user, the head-mounted camera 304 is locatedto the right of the lateral canthus of the eye, and eye positions invicinity of the medial canthus of the eye are outside the first range.For example, the eye positions in vicinity of the medial canthus, whichare outside the first range, may span between 10° and 50°.

The computer 308 calculates the eye positions based on the measurements303. The computer 308 then utilizes the calculated eye positions todetermine how to read images from the head-mounted camera 304. In oneembodiment, the computer 308 reads the images 305 from the head-mountedcamera 304 at a first bitrate when the eye positions fall within thefirst range. The computer 308 may refrain from reading any images fromthe head-mounted camera 304, when the eye positions fall outside thefirst range. Alternatively, the computer 308 may read the images 305from the head-mounted camera 304 at a second bitrate that is less thanhalf the first bitrate, when the eye positions fall outside the firstrange.

In one embodiment, the computer 308 calculates the eye positions basedon the measurements 303 at a rate that is at least ten times higher thana rate at which the images 305 are captured by the head-mounted camera304. Optionally, the computer 308 may command the head-mounted camera304 to operate in a low-power mode for a longer percent of time whilethe eye positions fall outside the first range, compared to a percent oftime it operates in low-power mode while the eye positions fall withinthe first range.

In normal operation, the computer 308 may command the head-mountedcamera 304 to operate in the low-power during at least some of the timesthe eye is in a position that falls outside the first range (e.g., aneye position that falls in the second range but not in the first range).The computer 308 may command the head-mounted camera 304 to exit thelow-power when the eye is at a position that is in the vicinity of thefirst range of eye positions. This may optimize the power consumption byallowing the head-mounted camera 304 to remain in the low-power modewhen the eye is not in a suitable position for the head-mounted camera304 to capture useful images, and/or when the user and or the eye are ina situation in which there is a reduced need for images captured by thehead-mounted camera 304. It is noted that if the head-mounted camera 304has its own computer to decide when to operate in low-power mode, thenthe computer 308 referred to in this paragraph comprises two or morecomputers: one in the head-mounted camera 304 and another that operatesthe head-mounted camera 304 in addition to operating other components.

Having multiple head-mounted cameras can be beneficial in someembodiments; multiple head-mounted cameras can be used to provide imagesof the eye from different perspectives and/or when the eye is indifferent ranges of positions. Having this additional data cancontribute, in some embodiments, to calculation of more accurate eyetracking values, such as gaze direction and/or pupil features and/orbeing able to calculate eye tracking values for a larger range of eyepositions.

In some embodiments, the eye tracking system may include a secondhead-mounted camera 306 that captures a second set of images 307 of theeye (which is captured in images 305) from a different position than thehead-mounted camera 304. Optionally, the computer 308 reads the secondset of images 307 from the second head-mounted camera 306 at a higherbitrate when the eye positions fall outside the first range compared towhen the eye positions fall within the first range. Optionally, thehead-mounted camera 304 and the second head-mounted camera 306 arepositioned at least 2 cm apart horizontally.

In one embodiment, the computer 308 commands the head-mounted camera 304to operate in a low-power mode for a longer percent of time while theeye positions fall outside the first range compared to a percent of timeit operates in the low-power mode while the eye positions fall withinthe first range. Additionally, the computer 308 commands the secondhead-mounted camera 306 to operate in a low-power mode for a longerpercent of time while the eye positions fall within the first rangecompared to a percent of time it operates in the low-power mode whilethe eye positions fall outside the first range.

In another embodiment, the computer 308 may select which head-mountedcamera, from among the head-mounted camera 304 and the head-mountedcamera 306, to utilize based on an eye positions calculated based on themeasurements 303. Optionally, the computer 308 utilizes a lookup table,which designates for different eye positions, which of the head-mountedcameras to utilize in order to capture images of the eye. Optionally,the computer 308 may detect that a sensor shift has occurred, due to atleast one of the head-mounted camera 304 and the head-mounted camera 306being in a different position and/or orientation. For example, such ashift may occur when these cameras are coupled to smartglasses 300, andthe smartglasses 300 have moved due to the user removing them andputting them on again. Optionally, the sensor shift may be detectedbased on variations in the images 305 and/or the images 307 (e.g.,detecting that certain facial landmarks appear at a different locationand/or perspective). Additionally or alternatively, the sensor shift maybe detected based on variations in the measurements 303 (e.g., usingtechniques mentioned herein with respect to sensor shifts with PS OG).Upon detecting the sensor shift, the computer 308 may update the lookuptable, to reflect possible changes to ranges of eye positions bestsuited to be captured with each of the head-mounted cameras. Forexample, the computer 308 may change co-ordinates of the ranges (andwhich eye positions fall within them), based on specific parametersdetermined for the sensor shift (i.e., what change in location and/ororientation was determined for the sensor shift).

FIG. 9B illustrates an embodiment in which multiple head-mounted camerasare used for eye tracking. The figure illustrates a portion of asmartglasses frame that is near the right eye of a user. The frame hasembedded in it discrete photosensors (such as photosensor 302 a) andemitters (such as emitter 302 b) that provide measurements from whichthe computer 308 may determine the eye positions. The frame also has twohead-mounted cameras coupled to it. The head-mounted camera 306 islocated to the right of the eye and the head-mounted camera 304 islocated to the left of the eye. The head-mounted camera 306 is betterpositioned to capture images of the user's pupil when the eye looks tothe right, and the head-mounted camera 304 is better positioned tocapture images of the user's pupil when the eye looks to the left.

In some embodiments, the head-mounted camera 304 and the secondhead-mounted camera 306 may be connected to the computer 308 over a busthat does not have sufficient bandwidth to transfer images from bothcameras at a maximum bitrate at which the computer 308 is capable ofreading from each of the cameras. Thus, the computer 308 may read atcertain times, images at a higher bitrate from the camera that is betterpositioned to capture the pupil at a specific time.

In some embodiments, due to the positioning of the head-mounted camera304 (e.g., when coupled to the frame of the smartglasses 300), there maybe an acute angle between the focal plane of the head-mounted camera 304and a plane representing the eye (e.g., the plane of the iris of theeye). Having an acute angle between these planes (e.g., an angle of lessthan 30°) may result in images captured by the head-mounted camera 304being unclear. One corrective approach, which may be utilized in someembodiments, is to utilize the Scheimpflug principle, and have thehead-mounted camera 304 configured such that its sensor is shiftedrelative to its optics (i.e., the focal plane). Optionally, thehead-mounted camera 304 includes a mechanism that enables it to tiltand/or shift its sensor and/or its optics. For example, tilting and/orshifting the sensor and/or the optics of the head-mounted camera 304 maybe done utilizing a piezoelectric motor and/or a microelectromechanicalsystem (MEMS). Optionally, the computer 308 commands the head-mountedcamera 304 to perform the tilting and/or shifting according to the eyepositions. Optionally, the tilting and/or shifting based on the eyepositions are done in order to obtain a certain angle between the sensorplane and the focal plane of the camera which is suitable forcorrecting, according to the Scheimpflug principle, image aberrationsdue to the acute angle between the head-mounted camera 304 and the irisplane of the eye when the eye is in a certain eye position. Optionally,the extent of the shifting and/or tilting performed when the eye isdetected in the certain eye position is a predetermined according tocalculations performed according to the Scheimpflug principle, which areknown in the art. Optionally, the tilting and/or shifting comprises atleast one of: tilting the sensor and/or the optics by an angle that isbetween 2.0° and 9° , and shifting the sensor and/or the optics by 1 to3 mm.

In some embodiments, the head-mounted device 302 may include one or moredevices of one or more types. In one embodiment, the head-mounted device302 includes an electrooculography device, and the measurements 303 areindicative of electrical potentials between electrodes placed close tothe eye. In another embodiment, the head-mounted device 302 includes anelectromyography device, and the measurements 303 are indicative ofelectrical potentials generated by muscle cells. In another embodiment,the head-mounted device 302 includes an optical flow sensor, themeasurements 303 are indicative of optical flow and/or visual motion,and the eye position is calculated using an optical flow algorithm. Instill another embodiment, the head-mounted device 302 includes rangesensors, and the measurements 303 are indicative of ranges between therange sensors and the eye.

In some embodiments, the head-mounted device 302 includes aphotosensor-oculography device (PSOG), and the measurements 303 are ofreflections of light emitted by the PSOG towards the eye. In someembodiments, designing the PSOG to have a wider tracking range comparedto the first range trackable from head-mounted camera 304 can lead to acost reduction. It may also lead to a weight reduction by enabling thesystem to capture each eye with a single camera located closer to theface than it would have been needed to be located had the head-mountedcamera 304 had to capture a wider range of eye positions. In addition,the computer 308 can save power by lowering utilization of thehead-mounted camera 304 when the eye is at an eye position in the secondrange (and not in the first range) compared to the utilization of thehead-mounted camera 304 when the eye is at a position that falls in thefirst range.

In one embodiment, in which the head-mounted device 302 includes PSOG,the computer 308 utilizes a model to calculate the eye positions basedon the measurements 303, which in this embodiment comprise measurementsof reflections from the eye. As discussed elsewhere herein, e.g., in thediscussion regarding the PSOG 235, this model may benefit fromcalibration. Optionally, calibration of the model may be done based onimages captured by the head-mounted camera 304 when the first eyepositions fall within the first range. Optionally, the model may becalibrated for eye positions that fall in the second range byextrapolating changes made to the model with eye positions in the firstrange. Optionally, the model may be calibrated for detecting eyepositions that fall in the second range in the factory, or with anon-head-mounted VOG when the system has clear lenses that do notdisturb the ability of the non-head-mounted VOG to collect the groundtruth measurements required to calibrate the PSOG model.

In another embodiment, in which the head-mounted device 302 includesPSOG, the computer 308 utilizes a model to calculate the eye positionsbased on the measurements 303, which in this embodiment comprisemeasurements of reflections from the eye. Additionally, the computer 308may utilize the images 305 (along with the measurements of thereflections) to calculate the eye positions. For example, utilizing bothsources of data may result in more accurate calculations of eyepositions than attainable when relying on only one of the sources ofdata. To calculate eye positions based on both sources of data, thecomputer 308 generates feature values based on data comprisingmeasurements of the reflections and the images 305, utilizes a model tocalculate the eye positions based on the feature values. Optionally, thefeature values utilized are feature values described herein for the taskof determining eye positions with PSOG and/or feature values known inthe art for determining eye positions based on images.

In yet another embodiment, in which the head-mounted device 302 includesPSOG, the computer 308 extracts a set of eyelid positions based onanalyzing the images 305. These extracted eyelid positions, along withmeasurements of reflections of light from the eyes taken by the PSOG atthose times, can be utilized to train a model for calculating eyelidposition based on additional measurements of reflections of light fromthe eye (e.g., taken at a future time for which there are nocorresponding images). Optionally, the computer 308 generates trainingsamples, each comprising feature values that are based on the emittedlight and measurements of the reflections and a label based on theeyelid position at that time, and utilizes the training samples to trainthe model. Optionally, the computer 308 may generate feature values suchas the ones described herein as being generated by the computer 278 forthe purpose of calculating eyelid position and/or training a model forthis purpose.

The following method may be used by some embodiments of systems modeledaccording to

FIG. 9A. The steps described below may be performed by running acomputer program having instructions for implementing the method.Optionally, the instructions may be stored on a computer-readablemedium, which may optionally be a non-transitory computer-readablemedium. In response to execution by a system including a processor andmemory, the instructions cause the system to perform the followingsteps:

In Step 1, taking, with a head-mounted device (e.g., the head-mounteddevice 302), measurements indicative of positions of an eye of a user(referred to herein as “eye positions”).

In Step 2, capturing images of the eye by a head-mounted camera (e.g.,the head-mounted camera 304). Optionally, a first range of eye positionstrackable from the images is narrower than a second range of eyepositions trackable from the measurements taken in Step 1.

In Step 3, calculating eye positions based on the measurements taken inStep 1.

In Step 4, reading the images from the head-mounted camera at a firstbitrate when the eye positions fall within the first range.

And in Step 5, refraining from reading the images from the head-mountedcamera, or reading the images from the head-mounted camera at a secondbitrate that is less than half the first bitrate, when the eye positionsfall outside the first range.

In one embodiment, the method may optionally include a step ofcommanding the head-mounted camera to operate in a low-power mode for alonger percent of time while the eye positions fall outside the firstrange, compared to a percent of time it operates in low-power mode whilethe eye positions fall within the first range.

In one embodiment, the method may optionally include the followingsteps: capturing, by a second head-mounted camera, a second set ofimages of the eye from a different position than the head-mountedcamera; and reading the second set of images from the secondhead-mounted camera at a higher bitrate when the eye positions falloutside the first range compared to when the eye positions fall withinthe first range.

The eye tracking system illustrated in FIG. 9A may be utilized in someembodiments to calculate pupil features based on the images 305, whichare captured by the head-mounted camera 304.

In some embodiments, an eye tracking system includes the head-mounteddevice 302, which is configured to take measurements 303 indicative ofpositions of an eye of a user (referred to herein as “eye positions”)and the head-mounted camera 304. In these embodiments, the head-mountedcamera 304 is better positioned to capture images of the user's pupilwhen the eye positions fall within a certain range compared to when theeye positions fall outside the certain range. In some examples, thecertain range may be the first range mentioned further above, of eyepositions that are trackable from the images 305. In one example, theeye positions that fall outside the certain range span at least 20° ofthe eye's field of view. In another example, the eye positions that falloutside the certain range span at least 40° of the eye's field of view.

The computer 308 may calculate, in some embodiments, pupil features 309based on images captured by the head-mounted camera 304. Optionally, thepupil features 309 include at least one of: a pupil center location, apupil contour, a pupil diameter, a glint location, a glint-pupil vector,a Haar-like feature, a result of an ellipse fitting approach, and aresult of pupil edge filtering. The aforementioned pupil features arewell known in the art, and discussed in the reference Fuhl, Wolfgang, etal. “Pupil detection for head-mounted eye tracking in the wild: anevaluation of the state of the art”, Machine Vision and Applications27.8 (2016): 1275-1288, which is incorporated herein by reference.

The extent to which the pupil features 309 are calculated by thecomputer 308 can depend on the eye positions (which are calculated basedon the measurements 303). In one embodiment, the computer 308calculates, at a first average rate, pupil features 309 based on imagescaptured by the head-mounted camera 304 when the eye positions fallwithin the certain range. For example, the computer 308 may calculatethe pupil features 309 at the first rate based on the images 305 (whenthe first range discussed further above is the same as the certainrange). Additionally, the computer 308 may refrain altogether fromcalculating pupil features when the eye positions fall outside of thecertain range. Alternatively to refraining from calculating pupilfeatures, the computer 308 may calculate pupil features at a secondaverage rate, which is less than half the first average rate, based onimages captured when the eye positions fall outside the certain range.Optionally, the first average is at least ten times the second averagerate. In one example, the first average rate is 100 Hz and the secondaverage rate is 5 Hz. In another example, the first average rate is 200Hz or more and the second average rate is below 20 Hz.

In some embodiments, the computer 308 commands the head-mounted camera304 to operate in a low-power mode for a longer percent of time whilethe eye positions fall outside the certain range, compared to a percentof time it operates in the low-power mode while the eye positions fallwithin the certain range.

In some embodiments, calculating the pupil features 309 based on themeasurements 303 is performed at a rate that is at least ten timeshigher than a rate at which the images 305 are captured by thehead-mounted camera 304.

In some embodiments, the second head-mounted camera 306 captures asecond set of images of the eye from a different position than thehead-mounted camera 304. Optionally, the computer 308 calculates pupilfeatures based on the second set of images at a higher rate when the eyepositions fall outside the certain range compared to when the eyepositions fall within the certain range. Optionally, the computer 308commands the second head-mounted camera 306 to operate in a low-powermode for a longer percent of time while the eye positions fall withinthe certain range, compared to a percent of time it operates in thelow-power mode while the eye positions fall outside the certain range.

The following method may be used by some embodiments of systems modeledaccording to FIG. 8 . The steps described below may be performed byrunning a computer program having instructions for implementing themethod. Optionally, the instructions may be stored on acomputer-readable medium, which may optionally be a non-transitorycomputer-readable medium. In response to execution by a system includinga processor and memory, the instructions cause the system to perform thefollowing steps:

In Step 1, taking, with a head-mounted device (e.g., the head-mounteddevice 302), measurements indicative of positions of an eye of a user(referred to herein as “eye positions”).

In Step 2, capturing images of the eye by a head-mounted camera (e.g.,the head-mounted camera 304).

In Step 3, calculating eye positions based on the measurements taken inStep 1. Optionally, the head-mounted camera used in Step 2 is betterpositioned to capture images of the user's pupil when the eye positionsfall within a certain range compared to when the eye positions falloutside the certain range.

In Step 4, calculating, at a first average rate, pupil features based onimages captured when the eye positions fall within the certain range.

And in Step 5, refraining from calculating pupil features, orcalculating pupil features at a second average rate that is less thanhalf the first average rate, based on images captured when the eyepositions fall outside the certain range.

In one embodiment, calculating of the eye positions based on themeasurements in Step 3 is performed at a rate that is at least ten timeshigher than rate of capturing the images by the head-mounted camera.Optionally, the method also includes a step of commanding thehead-mounted camera to operate in a low-power mode for a longer percentof time while the eye positions fall outside the certain range comparedto a percent of time it operates in low-power mode while the eyepositions fall within the certain range.

In one embodiment, the method also includes the following steps:capturing, by a second head-mounted camera, a second set of images ofthe eye from a different position than the head-mounted camera, andcalculating pupil features based on the second set of images at a higherrate when the eye positions fall outside the certain range compared towhen the eye positions fall within the certain range. Optionally, themethod also includes the following steps related to power management:commanding the head-mounted camera to operate in a low-power mode for alonger percent of time while the eye positions fall outside the certainrange compared to a percent of time it operates in the low-power modewhile the eye positions fall within the certain range; and commandingthe second head-mounted camera to operate in a low-power mode for alonger percent of time while the eye positions fall within the certainrange compared to a percent of time it operates in the low-power modewhile the eye positions fall outside the certain range.

FIG. 10 illustrates an embodiment of an eye tracking system thatinvolves operation of a camera according to eyelid position. In oneembodiment, the eye tracking system includes the head-mounted device322, an inward-facing head-mounted camera 324, and a computer 328. Theinward-facing head-mounted camera 324 captures images 326 of an eye. Thehead-mounted device 322 emits electromagnetic waves towards the eye, andtakes measurements 323 of reflections of the electromagnetic waves fromthe eye. Optionally, an average rate at which the reflections aremeasured is at least ten times an average rate at which the images 326are captured.

The head-mounted device 322 may include different types of components inembodiments described herein. These different components may providedifferent types of signals indicative of the position of the eyelids.

In one embodiment, the head-mounted device 322 includes aphotosensor-oculography device (PSOG) that measures reflections of lightemitted by the PSOG towards the eye. It is to be noted that thenear-infrared (NIR) reflectance of the eyelid is different from the NIRreflectance of the eye, and the NIR reflectance of the eyelashes isdifferent from the NIR reflectance of the skin. Thus, when thehead-mounted device 322 includes PSOG, movements of the eyelid changethe measured NIR reflections, thus enabling detection of the eyelidposition (based on changes to the measured reflections, which arecharacteristic of the eyelid in different positions).

In another embodiment, the head-mounted device 322 includes an opticalflow sensor that measures optical flow and/or visual motion, and theeyelid position is calculated based on an optical flow algorithm. In oneexample, the optical flow sensor is an image sensor that measuresoptical flow and/or visual motion, which are affected by the position ofthe eyelid.

In yet another embodiment, the head-mounted device 322 includes rangesensors that measure ranges between the range sensors and the eye. Thesemeasurements are affected by the eyelid position, thus enablingdetection of the eyelid position (based on changes to the ranges, whichare characteristic of the eyelid in different positions).

The computer 328 detects, based on the measurements 323 of thereflections, a position of at least one of the upper and lower eyelidscovering the eye (referred to herein as “eyelid position”). The computer328 operates the head-mounted camera 324 according to the eyelidposition in a certain manner, which decreases, in the captured images326, areas depicting the at least one of the upper and lower eyelids,and optionally increases, in the captured images 326, areas depictingthe iris. Optionally, operating the head-mounted camera 324 in thecertain manner involves the computer 328 issuing commands 325 to thehead-mounted camera 324 based on the eyelid position.

Operating the head-mounted camera 324 in the certain manner, whichincreases the areas depicting the iris and decreases depiction of theeyelids in the captured images 326, may be done in different ways inembodiments described herein.

In one embodiment, the head-mounted camera 324 utilizes a sensor thatsupports changing of its region of interest (ROI), similar to thehead-mounted camera 274 described herein. In this embodiment, thecomputer 328 utilizes the eyelid position to set the ROI around aportion of the eye that is not covered by the at least one of upper andlower eyelids. One example of setting the ROI around a portion of theeye that is not covered by the eyelids is to select a rectangle having aheight that is a bit longer than the maximum distance between the upperand lower eyelids. In one example, setting the ROI around a portion ofthe eye that is not covered by the eyelids involves setting the ROI tocover less than half the sensor's field of view. In another example, thecomputer 328 sets the ROI such that the size of the ROI is proportionalto percent of the eyeball uncovered by the eyelid. The proportion may beimplemented in steps, such as having 2-6 predefined sizes for the ROI,which depend on the eyelid position. Alternatively, the proportion maybe finer, such that the more the at least one of upper and lower eyelidscovers the eyeball the smaller the size of the ROI.

FIG. 11A illustrates different ROIs that are set according to differenteyelid positions. The image on the left depicts an open eye in which theROI is large and includes the full pupil (with a bit of the upper andlower eyelids in the ROI. The middle image depicts a partially closedeye, in which the eyelids cover part of the pupil. The ROI in this imageis smaller than the one on the left due to the visible portion of thepupil being smaller The image on the right in FIG. 11A depicts ascenario in which the eye is shut and thus no ROI is set.

In another embodiment, in order to operate the head-mounted camera 324in the certain manner, which increases the areas depicting the iris anddecreases depiction of the eyelids in the captured images 326, thecomputer 328 refrains from commanding the head-mounted camera 324 tocapture images during eyeblinks and/or while the eye is closed. Thismanner of selectively commanding the head-mounted camera 324 isillustrated in FIG. 11B, in which the commands to capture images areissued when the eye is open or partially open (images on the left andcenter), but no command to capture is issued when the eye is closed(image on the right).

The eyelid position can be indicative of emotional responses of theuser. In some embodiments, when the eyelid is determined to be in acertain position and/or it is detected that the eyelid performs acertain movement, it can be advantageous to obtain additionalmeasurements in order to better ascertain the specifics of the user'semotional response occurring at that time. In one example, raising theupper eyelid may indicate an emotional response comprising an expressionof surprise, which may be worth monitoring. In this example, thecomputer 328 detects a raising of the upper eyelid based on themeasurements 323 of the reflections, and increases the frame rate of thehead-mounted camera 324 during the time the upper eyelid is raised abovea threshold. In another example, raising of the lower eyelid mayindicate an emotional response comprising an expression of anger, whichmay be worth monitoring. In this example, the computer 328 detects araising of the lower eyelid based on the measurements 323 of thereflections, and increases the frame rate of the head-mounted camera 324during the time the lower eyelid is raised above a threshold. To detectthe raising of the upper eyelid and/or the raising of the lower eyelid,the computer 328 may utilize a machine learning-based model, asdescribed in more detail below.

Various embodiments described herein involve detecting the eyelidposition (of the top eyelid and/or the bottom eyelid) based onmeasurements 323 of the reflections, and then utilizing the eyelidposition to determine operational parameters, such as issuing thecommands 325 to determine how the head-mounted camera 324 shouldoperate. In some embodiments, the eyelid position can often more easilyand/or accurately be determined from images (e.g., images taken by thehead-mounted camera 324), using image analysis techniques known in theart. However, as explained below, by using such images to train amachine learning-based model, it is possible to determine the eyelidposition, even at times at which images are not captured by thehead-mounted camera 324, which can help conserve power.

In some embodiments, measurements of the reflections obtained by thehead-mounted device 322 are used to detect the eyelid position bycalculating, based on the measurements of the reflections, a valueindicative of the eyelid position. The calculation of the valueindicative of the eyelid position may utilize, in these embodiments, amodel that is trained with samples that each includes feature valuesthat are generated based on measurements of reflections (measured by thehead-mounted device 322) and a label generated from analysis of one ormore images captured by a certain camera at the same time themeasurements used to generate the feature values were taken. In oneexample, the certain camera may be the head-mounted camera 324. Inanother example, the certain camera may be a non-head-mounted device,such as a camera of a cellphone, webcam, laptop, or some othernon-head-mounted camera. Examples of the feature values that may begenerated by the computer 328 include values indicating the level ofelectromagnetic waves emitted towards the eye and the extent ofreflections.

In one example, in which the head-mounted device 322 includes PSOG atleast some of the feature values may include various PSOG-relatedfeature values that are known in the art mentioned in referencesdescribed herein. In another example, at least some of the featurevalues may include values indicative of the extent of optical flowand/or changes to the extent of optical flow, which are known in theart, as discussed in Jiménez-Pinto, et al., “Optical flow and driver'skinematics analysis for state of alert sensing.” Sensors 13.4 (2013):4225-4257 and in Cho, et al., “Moving object detection based on opticalflow estimation and a Gaussian mixture model for advanced driverassistance systems”, in Sensors 19.14 (2019): 3217. In yet anotherexample, in which the head-mounted device 322 includes range sensors, atleast some the feature values are indicative of the ranges measuredbetween the ranges sensors and the eye.

The labels used for the training samples may be generated using imageanalysis techniques known in the art for determining positions ofeyelids. Training samples that include the aforementioned feature valuesand labels are provided to a machine learning algorithm, such as analgorithm for training a regression model, a neural network, etc. Thisresults in a personalized model that is capable of detecting the eyelidposition of the user based on measurements of the user obtained with thehead-mounted device 322.

In one example, the computer 328 extracts eyelid positions by analyzingthe images 326 (referred to herein as “image-based positions”), andutilizes the image-based positions and the measurements 323 of thereflections to generate labeled training samples for training a machinelearning-based model for detecting the eyelid position based onadditional measurements of reflections taken with the head-mounteddevice 322.

In another example, the computer 328 receives photos from a camerabelonging to a non-head-mounted device, extracts eyelid positions byanalyzing the photos (referred to herein as “photo-based positions”),and utilizes the photo-based positions and the measurements 323 of thereflections to generate labeled training samples for training a machinelearning-based model for detecting the eyelid position based onadditional measurements of reflections taken with the head-mounteddevice 322. In this example, the computer 328 may include a head-mountedcomputer and a non-head-mounted computer that are configured tocommunicate over a wireless communication channel, the non-head-mountedcomputer is used to train the model, and the head-mounted computer mayuse the trained model for detecting the eyelid position based onmeasurements of the reflections measured by the head-mounted device 322.

In one embodiment, the computer 328 calculates a level of tiredness ofthe user based on the eyelid position, and adjusts a frequency at whichthe images 326 are captured based on the level of tiredness. The levelof tiredness can be used in various applications. For example, whileusing an entertainment application, the computer 328 may reduce thefrequency at which the images 326 are captured as the user becomes moretired because it may be assumed that the user's responses become slower.In another example, while using a driver safety application, thecomputer 328 may increase the frequency at which the images are capturedas the user becomes more tired.

Detecting a level of tiredness may be done utilizing a machinelearning-based model, which is provided feature values generated basedon data that includes indications of the eyelid position (and possiblyother data indicative of the physiological state of the user and/or theactivity level of the user). The computer 328 may then utilize a certainmodel to calculate, based on the feature values, a value indicative ofthe tiredness level of the user. The certain model, in this embodiment,may be trained with training samples, each including: (i) feature valuesgenerated based on measurements of a certain user at a certain time(which include an indication of the eyelid position at the certain timeand possibly other data indicative of the physiological state of thecertain user and/or the activity level of the certain user at thecertain time), and (ii) a label indicative of the tiredness level of thecertain user at the certain time. Some examples of feature values thatmay be determined based on the eyelid position include averagerate/duration of number of blinks, average height of the eyelid, averagearea of the eye that is exposed, etc. The tiredness level values of thecertain user (which are used in the training) may be determined invarious ways, such as a self-report by the certain user, or analysis ofimages of the certain user using image analysis methods known in the artof detecting the level of tiredness.

The following method may be used by some embodiments of systems modeledaccording to FIG. 10 . The steps described below may be performed byrunning a computer program having instructions for implementing themethod. Optionally, the instructions may be stored on acomputer-readable medium, which may optionally be a non-transitorycomputer-readable medium. In response to execution by a system includinga processor and memory, the instructions cause the system to perform thefollowing steps:

In Step 1, capturing images of an eye by an inward-facing head-mountedcamera (e.g., the head-mounted camera 324).

In Step 2, emitting, by a head-mounted device (e.g., the head-mounteddevice 322), electromagnetic waves towards the eye, and takingmeasurements of reflections of the electromagnetic waves from the eye(e.g., by the head-mounted device 322). Optionally, an average rate oftaking measurements of the reflections in this step is at least tentimes average rate of capturing the images in Step 1.

In Step 3, detecting, based on the measurements of the reflections, aposition of at least one of the upper and lower eyelids covering the eye(referred to herein as the “eyelid position”).

And in Step 4, operating the inward-facing head-mounted camera accordingto the eyelid position in a manner that (i) increases, in the capturedimages, areas depicting the iris, and (ii) decreases, in the capturedimages, areas depicting the at least one of the upper and lower eyelids.

In one embodiment, the inward-facing head-mounted camera utilizes asensor that supports changing of its region of interest (ROI), andoperating the inward-facing head-mounted camera in the manner describedin Step 4 involves utilizing the eyelid position to set the ROI around aportion of the eye that is not covered by the at least one of upper andlower eyelids.

In one embodiment, the method optionally includes a step involvingcalculating a level of tiredness based on the eyelid position, andadjusting a frequency at which the images are captured in Step 1 basedon the level of tiredness.

In one embodiment, the method optionally includes performing thefollowing steps to train a machine learning-based model that is used fordetecting the eyelid position based on additional measurements ofreflections (of electromagnetic waves from the eye): extracting eyelidpositions by analyzing the images (referred to herein as “image-basedpositions”); generating labels based on the image-based positions;generating feature values based on the measurements of the reflections;and providing the labels and feature values to train the machinelearning-based model for detecting the eyelid position based on theadditional measurements of the reflections. Optionally, the machinelearning-based model is used to detect the eyelid position at futuretimes during which images of the eye were not captured.

In one embodiment, the method optionally includes performing thefollowing steps to train a machine learning-based model that is used fordetecting the eyelid position based on additional measurements ofreflections (of electromagnetic waves from the eye): receiving photosfrom a camera belonging to a non-head-mounted device, extracting eyelidpositions by analyzing the photos (referred to herein as “photo-basedpositions”), generating labels based on the photo-based positions,generating feature values based on the measurements of the reflections,and providing the labels and feature values to train the machinelearning-based model for detecting the eyelid position based on theadditional measurements of the reflections. Optionally, the machinelearning-based model is used to detect the eyelid position at futuretimes during which images of the eye were not captured.

US Patent Application 2019/0223737A1, which is herein incorporated byreference in its entirety and is a previous patent application of theApplicant of this invention, discusses and illustrates in paragraphs0040-0049, together with their associated drawings, various examples ofhead-mounted systems equipped with head-mounted cameras, which can beadapted to be utilized with some of the embodiments herein. For example,these paragraphs illustrate various inward-facing head-mounted camerascoupled to an eyeglasses frame, illustrate cameras that capture regionson the periorbital areas, illustrate an optional computer that mayinclude a processor, memory, a battery and/or a communication module,illustrate inward-facing head-mounted cameras coupled to an augmentedreality devices, illustrate head-mounted cameras coupled to a virtualreality device, illustrate head-mounted cameras coupled to a sunglassesframe, illustrate cameras configured to capture various regions, such asthe forehead, the upper lip, the cheeks, and sides of the nose,illustrate inward-facing head-mounted cameras mounted to protrudingarms, illustrate various inward-facing head-mounted cameras havingmulti-pixel sensors (FPA sensors) configured to capture various regions,illustrate head-mounted cameras that are physically coupled to a frameusing a clip-on device configured to be attached/detached from a pair ofeyeglasses in order to secure/release the device to/from the eyeglasses,illustrate a clip-on device holds at least an inward-facing camera, aprocessor, a battery, and a wireless communication module, illustrateright and left clip-on devices configured to be attached behind aneyeglasses frame, illustrate a single-unit clip-on device configured tobe attached behind an eyeglasses frame, and illustrate right and leftclip-on devices configured to be attached/detached from an eyeglassesframe and having protruding arms to hold the inward-facing head-mountedcameras.

It is noted that the elliptic and other shapes of the regions capturedby cameras and other sensing devices (such as PSOG) in some of thedrawings are just for illustration purposes, and the actual shapes ofthe regions are usually not as illustrated. Furthermore, illustrationsand discussions of a camera represent one or more cameras, where eachcamera may have the same field of view (FOV) and/or different FOVs. Acamera includes multiple sensing elements, and the illustrated regioncaptured by the camera usually refers to the total region captured bythe camera, which is made of multiple regions that are respectivelycaptured by the different sensing elements. The positions of the camerasin the figures are just for illustration, and the cameras may be placedat other positions.

Various embodiments described herein involve a head-mounted system (HMS)that may be connected, using wires and/or wirelessly, with a devicecarried by the user and/or a non-wearable device. The HMS may include abattery, a computer, sensors, and a transceiver.

FIG. 12A and FIG. 12B are schematic illustrations of possibleembodiments for computers (400, 410) that are able to realize one ormore of the embodiments discussed herein that include a “computer”. Thecomputer (400, 410) may be implemented in various ways, such as, but notlimited to, a microcontroller, a computer on a chip, a system-on-chip(SoC), a system-on-module (SoM), a processor with its requiredperipherals, a server computer, and/or any other computer form capableof executing a set of computer instructions. Further, references to acomputer or a processor include any collection of one or more computersand/or processors (which may be at different locations) thatindividually or jointly execute one or more sets of computerinstructions. This means that the singular term “computer” is intendedto imply one or more computers, which jointly perform the functionsattributed to “the computer”. In particular, some functions attributedto the computer may be performed by a computer on a wearable device(e.g., smartglasses) and/or a computer of the user (e.g., smartphone),while other functions may be performed on a remote computer, such as acloud-based server.

The computer 400 includes one or more of the following components:processor 401, memory 402, computer readable medium 403, user interface404, communication interface 405, and bus 406. The computer 410 includesone or more of the following components: processor 411, memory 412, andcommunication interface 413.

Functionality of various embodiments may be implemented in hardware,software, firmware, or any combination thereof. If implemented at leastin part in software, implementing the functionality may involve acomputer program that includes one or more instructions or code storedor transmitted on a computer-readable medium and executed by one or moreprocessors. Computer-readable media may include computer-readablestorage media, which corresponds to a tangible medium such as datastorage media, and/or communication media including any medium thatfacilitates transfer of a computer program from one place to another.Computer-readable medium may be any media that can be accessed by one ormore computers to retrieve instructions, code, data, and/or datastructures for implementation of the described embodiments. A computerprogram product may include a computer-readable medium. In one example,the computer-readable medium 403 may include one or more of thefollowing: RAM, ROM, EEPROM, optical storage, magnetic storage, biologicstorage, flash memory, or any other medium that can store computerreadable data.

A computer program (also known as a program, software, softwareapplication, script, program code, or code) can be written in any formof programming language, including compiled or interpreted languages,declarative or procedural languages. The program can be deployed in anyform, including as a standalone program or as a module, component,subroutine, object, or another unit suitable for use in a computingenvironment. A computer program may correspond to a file in a filesystem, may be stored in a portion of a file that holds other programsor data, and/or may be stored in one or more files that may be dedicatedto the program. A computer program may be deployed to be executed on oneor more computers that are located at one or more sites that may beinterconnected by a communication network.

Computer-readable medium may include a single medium and/or multiplemedia (e.g., a centralized or distributed database, and/or associatedcaches and servers) that store one or more sets of instructions. Invarious embodiments, a computer program, and/or portions of a computerprogram, may be stored on a non-transitory computer-readable medium, andmay be updated and/or downloaded via a communication network, such asthe Internet. Optionally, the computer program may be downloaded from acentral repository, such as Apple App Store and/or Google Play.Optionally, the computer program may be downloaded from a repository,such as an open source and/or community run repository (e.g., GitHub).

At least some of the methods described herein are “computer-implementedmethods” that are implemented on a computer, such as the computer (400,410), by executing instructions on the processor (401, 411).Additionally, at least some of these instructions may be stored on anon-transitory computer-readable medium.

As used herein, references to “one embodiment” (and its variations) meanthat the feature being referred to may be included in at least oneembodiment of the invention. Separate references to embodiments mayrefer to the same embodiment, may illustrate different aspects of anembodiment, and/or may refer to different embodiments.

Sentences in the form of “X is indicative of Y” mean that X includesinformation correlated with Y, up to the case where X equals Y.Sentences in the form of “provide/receive an indication (of whether Xhappened)” may refer to any indication method.

The word “most” of something is defined as above 51% of the something(including 100% of the something). Both a “portion” of something and a“region” of something refer to a value between a fraction of thesomething and 100% of the something The word “region” refers to anopen-ended claim language, and a camera said to capture a specificregion on the face may capture just a small part of the specific region,the entire specific region, and/or a portion of the specific regiontogether with additional region(s). The phrase “based on” indicates anopen-ended claim language, and is to be interpreted as “based, at leastin part, on”. Additionally, stating that a value is calculated “based onX” and following that, in a certain embodiment, that the value iscalculated “also based on Y”, means that in the certain embodiment, thevalue is calculated based on X and Y. Variations of the terms “utilize”and “use” indicate an open-ended claim language, such that sentences inthe form of “detecting X utilizing Y” are intended to mean “detecting Xutilizing at least Y”, and sentences in the form of “use X to calculateY” are intended to mean “calculate Y based on X”.

The terms “first”, “second” and so forth are to be interpreted merely asordinal designations, and shall not be limited in themselves. Apredetermined value is a fixed value and/or a value determined any timebefore performing a calculation that utilizes the predetermined value.When appropriate, the word “value” may indicate a “predetermined value”.The word “threshold” indicates a “predetermined threshold”, which meansthat the value of the threshold, and/or the logic used to determinewhether the threshold is reached, is known before start performingcomputations to determine whether the threshold is reached.

The embodiments of the invention may include any variety of combinationsand/or integrations of the features of the embodiments described herein.Although some embodiments may depict serial operations, the embodimentsmay perform certain operations in parallel and/or in different ordersfrom those depicted. Moreover, the use of repeated reference numeralsand/or letters in the text and/or drawings is for the purpose ofsimplicity and clarity and does not in itself dictate a relationshipbetween the various embodiments and/or configurations discussed. Theembodiments are not limited in their applications to the order of stepsof the methods, or to details of implementation of the devices, set inthe description, drawings, or examples. Moreover, individual blocksillustrated in the figures may be functional in nature and therefore maynot necessarily correspond to discrete hardware elements.

Certain features of the embodiments, which may have been, for clarity,described in the context of separate embodiments, may also be providedin various combinations in a single embodiment. Conversely, variousfeatures of the embodiments, which may have been, for brevity, describedin the context of a single embodiment, may also be provided separatelyor in any suitable sub-combination. Embodiments described in conjunctionwith specific examples are presented by way of example, and notlimitation. Moreover, it is evident that many alternatives,modifications, and variations will be apparent to those skilled in theart. It is to be understood that other embodiments may be utilized andstructural changes may be made without departing from the scope of theembodiments. Accordingly, this disclosure is intended to embrace allsuch alternatives, modifications, and variations that fall within thespirit and scope of the appended claims and their equivalents.

We claim:
 1. An eye tracking system configured to operate a cameraaccording to eyelid position, comprising: an inward-facing head-mountedcamera configured to capture images of an eye; a head-mounted deviceconfigured to emit electromagnetic waves towards the eye, and to takemeasurements of reflections of the electromagnetic waves from the eye;wherein an average rate at which the measurements of the reflections aretaken is at least ten times an average rate at which the images arecaptured; and a computer configured to: detect, based on themeasurements of the reflections, a position of at least one of theeyelids covering the eye (eyelid position); and operate theinward-facing head-mounted camera according to the eyelid position in amanner that decreases, in the captured images, areas depicting the atleast one of the eyelids.
 2. The eye tracking system of claim 1, whereinthe inward-facing head-mounted camera utilizes a sensor that supportschanging of its region of interest (ROI), and the computer is furtherconfigured to utilize the eyelid position to set the ROI around aportion of the eye that is not covered by the at least one of theeyelids.
 3. The eye tracking system of claim 2, wherein the ROI coversless than half the sensor's field of view.
 4. The eye tracking system ofclaim 2, wherein the computer is configured to set the ROI such that thesize of the ROI is proportional to area of the eyeball uncovered by theeyelid.
 5. The eye tracking system of claim 2, wherein the computer isfurther configured to operate the inward-facing head-mounted cameraaccording to the eyelid position in a manner that increases, in thecaptured images, areas depicting the iris.
 6. The eye tracking system ofclaim 2, wherein the computer is further configured to detect a raisingof the upper eyelid based on the measurements of the reflections, and toincrease a frame rate of the inward-facing head-mounted camera duringthe time the upper eyelid is raised above a threshold; whereby raisingof the upper eyelid may indicate an emotional response comprising anexpression of surprise.
 7. The eye tracking system of claim 2, whereinthe computer is further configured to detect a raising of the lowereyelid based on the measurements of the reflections, and to increase aframe rate of the inward-facing head-mounted camera during the time thelower eyelid is raised above a threshold; whereby raising of the lowereyelid may indicate an emotional response comprising an expression ofanger.
 8. The eye tracking system of claim 1, wherein in order tooperate the inward-facing head-mounted camera according to the eyelidposition the computer is configured to refrain from commanding thecamera to capture the images during blinks and/or while the eye isclosed.
 9. The eye tracking system of claim 1, wherein the computer isfurther configured to: extract eyelid positions by analyzing the images(image-based positions), and utilize the image-based positions and themeasurements of the reflections to generate labeled training samples fortraining a machine learning-based model for detecting the eyelidposition based on additional measurements of reflections taken with thehead-mounted device.
 10. The eye tracking system of claim 1, wherein thecomputer is further configured to: receive photos from a camerabelonging to a non-head-mounted device, extract eyelid positions byanalyzing the photos (photo-based positions), and utilize thephoto-based positions and the measurements of the reflections togenerate labeled training samples for training a machine learning-basedmodel for detecting the eyelid position based on additional measurementsof reflections taken with the head-mounted device.
 11. The eye trackingsystem of claim 10, wherein the computer comprises a head-mountedcomputer and a non-head-mounted computer that are configured tocommunicate over a wireless communication channel; the non-head-mountedcomputer is configured to train the model, and the head-mounted computeris configured to use the trained model for detecting the eyelid positionbased on reflections.
 12. The eye tracking system of claim 1, whereinthe head-mounted device comprises a photosensor-oculography device(PSOG) configured to measure reflections of light emitted by the PSOGtowards the eye.
 13. The eye tracking system of claim 1, wherein thehead-mounted device comprises an optical flow sensor configured tomeasure optical flow and/or visual motion, and the eyelid position iscalculated based on an optical flow algorithm.
 14. The eye trackingsystem of claim 1, wherein the head-mounted device comprises rangesensors configured to measure ranges between the range sensors and theeye; whereby measurements of the range sensors are affected by theeyelid position.
 15. A method comprising: capturing images of an eye byan inward-facing head-mounted camera; emitting, by a head-mounteddevice, electromagnetic waves towards the eye, and taking measurementsof reflections of the electromagnetic waves from the eye; whereinaverage rate of taking measurements of the reflections is at least tentimes average rate of capturing the images; detecting, based on themeasurements of the reflections, a position of at least one of theeyelids covering the eye (eyelid position); and operating theinward-facing head-mounted camera according to the eyelid position in amanner that decreases, in the captured images, areas depicting the atleast one of the eyelids.
 16. The method of claim 15, wherein theinward-facing head-mounted camera utilizes a sensor that supportschanging of its region of interest (ROI), and further comprisingutilizing the eyelid position to set the ROI around a portion of the eyethat is not covered by the at least one of the eyelids.
 17. The methodof claim 15, further comprising operating the inward-facing head-mountedcamera according to the eyelid position in a manner that increases, inthe captured images, areas depicting the iris.
 18. The method of claim15, further comprising performing the following steps to train a machinelearning-based model that is used for detecting the eyelid positionbased on additional measurements of reflections: extracting eyelidpositions by analyzing the images (image-based positions); generatinglabels based on the image-based positions; generating feature valuesbased on the measurements of the reflections; and providing the labelsand feature values to train the machine learning-based model fordetecting the eyelid position based on the additional measurements ofthe reflections.
 19. The method of claim 15, further comprisingperforming the following steps to train a machine learning-based modelthat is used for detecting the eyelid position based on additionalmeasurements of reflections: receiving photos from a camera belonging toa non-head-mounted device, extracting eyelid positions by analyzing thephotos (photo-based positions), generating labels based on thephoto-based positions, generating feature values based on themeasurements of the reflections, and providing the labels and featurevalues to train the machine learning-based model for detecting theeyelid position based on the additional measurements of the reflections.20. A non-transitory computer readable medium storing one or morecomputer programs configured to cause a processor-based system toexecute steps comprising: capturing images of an eye by an inward-facinghead-mounted camera; emitting, by a head-mounted device, electromagneticwaves towards the eye, and taking measurements of reflections of theelectromagnetic waves from the eye; wherein average rate of takingmeasurements of the reflections is at least ten times average rate ofcapturing the images; detecting, based on the measurements of thereflections, a position of at least one of the eyelids covering the eye(eyelid position); and operating the inward-facing head-mounted cameraaccording to the eyelid position in a manner that decreases, in thecaptured images, areas depicting the at least one of the eyelids.